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Preface 


As  a  result  of  continuing  concern  with  large  cost  overruns  in  a  broad  range  of  major 
defense  programs,  Congress  enacted  new  statutory  provisions  extending  the  ambit  of 
the  existing  Nunn-McCurdy  Act.  In  accordance  with  the  revised  Nunn-McCurdy 
law,  the  Performance  Assessments  and  Root  Cause  Analysis  (PARCA)  office  must  pro¬ 
vide  its  root  cause  explanation  as  part  of  a  60-day  program  review  triggered  when  the 
breach  is  reported  by  the  applicable  military  department  secretary. 

In  March  2010,  the  newly  created  PARCA  office  within  the  Office  of  the  Sec¬ 
retary  of  Defense  (OSD),  in  view  of  staffing  limitations,  elected  to  rely  on  federally 
funded  research  and  development  center  (FFRDC)  support  to  help  discharge  its  new 
responsibilities.  It  engaged  the  RAND  Corporation  to  study  the  root  causes  of  Nunn- 
McCurdy  breaches  or  other  large  cost  increases  in  six  major  defense  acquisition  pro¬ 
grams:  the  Wideband  Global  Satellite,  the  Longbow  Apache,  the  DDG-1000,  the  Joint 
Strike  Fighter,  the  Excalibur,  and  the  Navy  Enterprise  Resource  Planning  (ERP). 

This  monograph  contains  the  analysis  performed  by  RAND  on  the  last  two 
of  these  six  root  cause  analyses:  Excalibur  and  the  Navy  ERP.  Analyses  of  the  other 
four  programs  appear  in  a  companion  report.1  In  addition,  this  report  develops  some 
exploratory  concepts  of  program  risk  and  complexity  as  factors  in  the  management  of 
program  acquisition.  This  report  should  interest  anyone  concerned  with  the  acquisition 
and  management  of  defense  systems. 

This  research  was  sponsored  by  OSD  PARCA  and  conducted  within  the  Acquisi¬ 
tion  and  Technology  Policy  Center  of  the  RAND  National  Defense  Research  Insti¬ 
tute,  a  federally  funded  research  and  development  center  sponsored  by  the  Office  of  the 
Secretary  of  Defense,  the  Joint  Staff,  the  Unified  Combatant  Commands,  the  Navy, 
the  Marine  Corps,  the  defense  agencies,  and  the  defense  Intelligence  Community. 

For  more  information  on  the  RAND  Acquisition  and  Technology  Policy  Center, 
see  http://www.rand.org/nsrd/ndri/centers/atp.html  or  contact  the  director  (contact 
information  is  provided  on  the  web  page). 


1  Irv  Blickstein  et  al.,  Root  Cause  Analyses  of  Nunn-McCurdy  Breaches,  Volume  1:  Zumwalt-C/dw  Destroyer, 
Joint  Strike  Fighter,  Longbow  Apache,  and  Wideband  Global  Satellite,  Santa  Monica,  Calif.:  RAND  Corporation, 
MG-1171/1-OSD,  2011. 
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Summary 


Background 

As  a  result  of  continuing  program  cost  growth  and  observations  by  the  Government 
Accountability  Office  (GAO)  placing  defense  acquisition  on  the  high-risk  target  list, 
Congress  became  particularly  concerned  about  the  execution  of  major  defense  acquisi¬ 
tion  programs.  This  concern  and  the  reality  of  shrinking  defense  budgets  led  Congress 
to  enact  laws  that  would  increase  the  focus  of  senior  policymakers  on  oversight  of 
major  defense  acquisition  programs  (MDAPs)  and  other  large  costly  programs.2  The 
Weapon  Systems  Acquisition  Reform  Act  (WSARA)  of  20093  established  a  number 
of  requirements  that  affected  the  operation  of  the  Defense  Acquisition  System  and 
the  duties  of  the  key  officials  who  support  it,  including  the  requirement  to  establish  a 
new  organization  in  the  Office  of  the  Secretary  of  Defense  (OSD)  with  the  mandate 
to  conduct  and  oversee  performance  assessments  and  root  cause  analysis  (PARCA)  for 
MDAPs. 

In  March  2010,  the  director  of  the  PARCA  office  determined  that  he  needed  sup¬ 
port  to  execute  his  statutory  responsibilities  and  turned  to  federally  funded  research 
and  development  centers  (FFRDCs)  and  academia  to  provide  that  support  for  the 
research  and  analysis  of  program  execution  status.  RAND  was  one  FFRDC  engaged 
to  perform  research  and  analysis  and  provide  recommendations  and  was  originally 
assigned  responsibility  for  four  programs.4  After  completing  that  initial  effort,  RAND 
was  assigned  two  additional  programs  for  research  and  analysis:  Excalibur  and  the 
Navy  Enterprise  Resource  Program  (ERP). 


Purpose 

This  report  does  two  things.  First,  it  analyzes  the  root  cause  of  cost  overruns  in  two 
programs:  the  Army  Excalibur  artillery  round  and  the  Navy  ERP.  The  Excalibur  proj- 

2  Ike  Skelton  Defense  Authorization  Act  for  Fiscal  Year  2011,  December  20,  2010. 

3  Public  Law  111-23,  Weapon  Systems  Acquisition  Reform  Act  of  2009,  May  22,  2009. 
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ect  incurred  a  Nunn-McCurdy  breach.  The  ERP  did  not,  but  the  cost  growth  was  so 
great  that  the  Department  of  Defense  (DoD)  requested  a  root  cause  analysis. 

Second,  it  presents  what  can  be  described  as  an  exercise  to  help  identify  the  most 
critical  features  of  a  program.  Critical  program  components  are  those  that  carry  the 
most  risk  of  overall  program  failure.  The  exercise  is  designed  to  identify  the  important 
program  features  that  decisionmakers  would  want  to  concentrate  on  when  inquiring 
about  a  program  as  it  develops  over  time.  The  report  then  uses  the  results  of  the  exercise 
to  flag  the  most  critical  features  of  the  weapon  system,  using  Excalibur  as  an  example. 
The  exercise  and  the  illustration  help  to  frame  one  approach  for  considering  program 
failure  risk  in  programs  that  have  not  yet  breached. 


Observations  on  the  Conduct  of  Root  Cause  Analyses 

Each  acquisition  program  is  unique,  and  each  root  cause  analysis  (RCA)  is  unique. 
However,  RAND’s  experience  in  conducting  six  root  cause  analyses  indicates  that  a  set 
of  core  activities  is  instrumental  to  a  successful  effort.  These  activities  define  a  generic 
root  cause  methodology  whose  key  components  include  the  following: 

•  Gather  and  review  readily  available  data. 

•  Develop  a  hypothesis. 

•  Set  up  long-lead-time  activities. 

•  Document  the  unit  cost  threshold  breach. 

•  Construct  a  time  line  of  relevant  cost  growth  events  in  the  program  history. 

•  Verify  the  cost  data  and  quantify  cost  growth. 

•  Create  and  analyze  the  program  cost  profiles  pinpointing  occurrences  of  cost 
growth. 

•  Match  the  time  line  events  with  changes  in  the  cost  profiles  and  derive  root  causes 
of  cost  growth. 

•  Reconcile  any  remaining  issues. 

•  Attribute  unit  cost  growth  to  root  causes. 

Successful  execution  of  this  set  of  activities  should  enable  the  research  team  to 
create  the  primary  deliverables  and  postulates  for  a  root  cause  analysis:  a  summary 
narrative  that  includes  clearly  stated  root  causes  of  cost  growth  supported  by  a  formal 
documentation  of  the  cost  threshold  breach,  a  summary  time  line  of  program  events 
leading  to  the  Nunn-McCurdy  breach,  funding  profiles,  a  completed  PARCA-office- 
generated  root  cause  matrix,  and  a  breakdown  of  the  amount  of  cost  growth  attrib¬ 
utable  to  each  root  cause;  a  briefing  that  corresponds  to  the  narrative;  and  a  full  root 
cause  report. 
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In  addition  to  developing  deliverables  and  postulates,  the  RCA  process  is  designed 
to  improve  the  research  focus  iteratively.  At  each  stage  of  the  RCA,  information  is 
drawn  from  and  contributed  to  the  program  archive.  The  RCA  analytic  team  can  use 
this  insight  not  only  to  improve  the  interim  products  that  result  from  successive  stages 
of  the  RCA  but  also  to  advance  the  original  hypothesis  that  guides  the  research.  This 
process  of  regularly  refining  the  guiding  hypothesis  with  the  insight  gained  during 
the  production  of  key  deliverables  and  postulates  enables  the  research  team  to  quickly 
identify  the  root  causes  of  a  program’s  failure. 


Findings  of  Root  Cause  Analyses 

Excalibur 

RAND’s  root  cause  analysis  identified  one  primary  driver  and  four  contributing  fac¬ 
tors  to  Excalibur’s  Nunn-McCurdy  cost  breach.  The  most  significant  source  of  cost 
increase  was  the  change  in  procurement  quantities',  a  79  percent  reduction  in  the  number 
of  Excalibur  rounds  ordered.  The  root  causes  of  this  quantity  reduction  were  changes 
in  requirements  combined  with  affordability  considerations.  Specifically,  the  manner 
in  which  artillery  was  being  used  and  the  precision  of  the  Excalibur  round  meant  that 
fewer  would  be  needed. 

An  Army  review  of  precision-guided  munitions  capability  placed  the  required 
quantity  of  Excalibur  munitions  in  the  context  of  the  other  guided  munitions  in  the 
Army’s  arsenal,  leading  to  a  decision  to  reduce  the  Army’s  procurement  objective  for 
Excalibur.  The  quantity  reduction,  resulting  from  changes  in  perceived  requirements, 
was  so  large  that  Nunn-McCurdy  unit  cost  breaches  would  have  occurred  even  in  the 
absence  of  any  other  factor. 

Another  four  factors  contributed  to  some  program  cost  growth  before  the  deci¬ 
sion  was  made  to  reduce  procurement  quantity.  Inaccurate  cost  estimates  contributed  to 
some  cost  growth.  Both  the  original  May  1997  cost  estimates  and  the  initial  Selected 
Acquisition  Report  (SAR)  estimates  were  too  low  to  reflect  the  technological  improve¬ 
ment  represented  by  Excalibur,  making  an  eventual  cost  overrun  more  likely.  Addi¬ 
tional  drivers  of  the  cost  growth  before  the  breach  include  a  concept  and  technologi¬ 
cal  change  that  occurred  between  the  original  solicitation  and  the  contract  award  in 
January  1998,  as  well  as  some  minor  technical  issues  that  were  identified  between  2002 
and  2010.  Finally,  Excalibur  unit  cost  growth  was  driven  by  the  validated  and  urgent 
operational  need for  Operation  Enduring  Freedom! Operation  Iraqi  Freedom  (OEF/OIF), 
which  caused  production  to  be  accelerated  and  more  Increment  la  rounds  produced 
than  initially  planned. 

Excalibur  was  unaffected  by  other  potential  root  causes.  For  example,  it  lived  up 
to  its  performance  expectations,  was  not  affected  by  poor  government  or  contractor 
performance,  and  had  sufficient  and  fairly  stable  sources  of  funding. 
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Navy  Enterprise  Resource  Planning 

Although  the  Navy  ERP  program  technically  breached  the  Nunn-McCurdy  cost 
growth  limits  and  was  implemented  behind  schedule,  the  program  can  be  considered 
a  qualified  success.  The  majority  of  cost  growth  and  schedule  delays  occurred  in  2004 
and  2005.  Since  the  2006  re-baseline,  costs  have  stabilized  and  production  delays  have 
been  limited. 

Part  of  the  root  cause  of  the  2004  cost  overrun  was  a  somewhat  optimistic  baseline 
for  cost  and  schedule.  The  greater  problem  was  the  unexpected  change  in  business  prac¬ 
tices  caused  by  the  Navy’s  decision,  after  the  Base  Realignment  and  Closure  (BRAC) 
process,  to  move  maintenance  from  an  intermediate-level  construct  to  a  regional  one. 
The  latter  led  to  the  major  schedule  slippage  in  2005  and  forced  the  ERP  program  to 
jettison  its  extension  to  maintenance  activities. 

The  ERP  program  was  re-baselined  in  2006  at  $400  million  higher.  The  increase 
arose  from  a  redesign  of  the  system,  a  change  in  business  practices,  and  an  improve¬ 
ment  in  estimates.  Since  2006,  ERP  costs  have  stabilized  and  the  program  has  been 
successfully  implemented  at  three  System  Commands  (SYSCOMs).  Minor  additional 
slippage  in  schedule  has  occurred  primarily  as  a  result  of  timing  issues  rather  than  pro¬ 
gram  delays  or  failures. 

The  Navy  ERP  can  be  considered  a  qualified  success.  Although  initial  cost  growth 
and  schedule  slippage  were  significant,  they  were  not  explosive,  and  the  ERP  program 
was  never  in  real  danger.  Several  factors  may  have  contributed  to  relative  program  suc¬ 
cess,  including  the  use  of  pilot  projects,  cost-plus  contracting,  the  decision  to  minimize 
the  customization  of  the  SAP  solution,  interactive  governance  and  high  leadership 
interest,  and  a  willingness  to  rely  on  the  managerial  and  technical  expertise  of  civilian 
cadres. 


Program  Complexity 

One  conclusion  drawn  from  the  analysis  of  the  six  programs  that  had  Nunn-McCurdy 
breaches  is  that  key  decisionmakers  lacked  adequate  visibility  into  the  programs.  After 
analyzing  the  programs  that  breached,  it  became  clear  that  indications  existed  that  a 
breach  was  possible  (or  even  likely),  but  they  were  buried  in  the  program  documen¬ 
tation.  This  opaqueness  occurs  because  key  details  can  be  hidden  in  the  voluminous 
documents  a  program  produces  or  can  appear  only  as  elliptical  references  in  program 
reports.  Thinking  about  how  to  mitigate  this  problem,  RAND  researchers  determined 
that  a  well-constructed  framework  could  help  decisionmakers  identify  areas  where  a 
program  might  have  greater  risk  for  problems  (and  thus  a  potential  Nunn-McCurdy 
breach)  so  that  they  could  direct  more  management  attention  to  those  areas. 

The  research  team  proposed  that  decisionmakers  use  a  “selective  screening  of  criti¬ 
cal  components”  process  to  identify  the  features  of  most  risk  to  a  program.  The  process 
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relates  measures  of  merit  drawn  from  a  variety  of  Jane’s  publications  used  to  describe 
programs  to  the  complexity  and  level  of  data  detail  available  for  specific  program  fea¬ 
tures.  A  measure  of  merit  is  broadly  a  set  of  technical  components  that  contribute  to 
a  measurable  process.  An  example  of  a  measure  of  merit  pertains  to  the  turboshafts 
of  the  Apache  helicopter  and  is  described  as  the  “maximum  continuous  drive”  for 
the  platform.  Other  helicopters  and  other  systems  also  use  the  maximum  continuous 
drive  measure  of  merit.  The  measure  includes  specific  technical  components  as  well  as 
systems  of  components  used  to  generate  a  particular  level  of  performance,  in  this  case 
maximum  continuous  drive.  Researchers  developed  a  graphical  display  technique  to 
help  identify  likely  areas  of  risk. 

The  most  important  measures  of  merit  for  program  personnel  to  consider  are  the 
ones  that  are  both  highly  complex  and  the  least  visible.  The  display  in  Figure  S.l  shows 
measures  of  merit  that  have  been  coded  for  level  of  complexity  and  detail  required  (e.g., 
a  more  complex  system  requires  more  detail). 

With  this  tool,  the  decisionmaker  or  analyst  can  evaluate  the  frequency  of  com¬ 
ponents  at  various  regions  of  the  resultant  “complexity-detail  matrix”  to  get  a  better 
view  of  the  measures  of  merit  that  contain  the  program  features  with  the  most  poten¬ 
tial  risk.  Construction  of  this  matrix  is  an  important  aspect  of  the  selective  screening 
process.  For  the  Longbow  Apache,  the  shaded  blue  square  highlights  the  system  com¬ 
ponents  that  are  closer  to  the  upper  right  corner  of  the  display,  i.e.,  those  that  are  the 
most  complex  yet  the  least  visible.  These  are  the  ones  that  warrant  greater  attention 

Figure  S.l 
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from  the  program  managers.  Use  of  common  metrics  allows  programs  to  be  compared 
across  systems. 


Program  Risk 

The  project  team  also  developed  a  methodology  to  identify  technical  risk  in  a  pro¬ 
gram’s  most  critical  components  early  enough  to  allow  project  managers  to  take  action 
to  avoid  a  Nunn-McCurdy  breach.  The  risk  experiment  explored  the  Excalibur  artil¬ 
lery  round.  First,  researchers  went  through  the  process  described  above  to  identify  the 
key  components,  i.e.,  those  on  the  critical  path  of  program  success.  For  Excalibur,  these 
turned  out  to  be  the  global  positioning  system  (GPS)  and  the  inertial  measurement 
unit  (IMU). 

Having  identified  the  critical  components  of  the  Excalibur  program,  the  team  then 
turned  to  Defense  Contract  Management  Agency  (DCMA)  parts  management  pro¬ 
gram  (PMP)  and  Defense  Acquisition  Executive  Summary  (DAES)  risk  assessments  to 
ascertain  if  either  DCMA  or  DAES  presaged  the  problems.  The  DCMA  reports  were 
issued  monthly.  Those  RAND  received  covered  only  31  months  of  a  13-year  program, 
but  they  contained  enough  data  to  detect  patterns.  The  DCMA  reports  use  a  stoplight 
system  to  highlight  risk  for  technical  components.  The  DAES  risk  assessments  are  peri¬ 
odic  summaries  provided  to  the  Defense  Acquisition  Executive. 

Review  of  the  DCMA  reports  showed  that  the  summary-level  judgments  assessed 
moderate  program  risk,  but  delving  into  the  data  at  the  subcategory  level  uncovered 
a  different  picture.  Arraying  the  lower-level  DCMA  component  ratings  against  the 
DAES  summary  ratings  showed  that  although  the  DAES  ratings  never  changed  from 
moderate  risk,  the  DCMA  component  ratings  showed  numerous  instances  when  risk 
ratings  for  the  IMU  were  rated  as  high.  Yet  the  fact  that  a  component  essential  to  the 
success  of  the  program  was  seen  as  high  risk  because  of  cracks  in  components  revealed 
during  testing  over  several  rating  periods  was  not  brought  to  the  attention  of  senior 
decisionmakers.  The  GPS  receiver  also  experienced  problems,  with  communications 
and  software  in  this  instance,  which  caused  several  flight  failures.  Better  use  of  avail¬ 
able  data  could  have  signaled  potential  problems  to  senior  program  personnel. 
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Introduction 


Background 

The  U.S.  Congress  continues  to  express  concern  with  cost  increases  in  major  defense 
acquisition  programs  (MDAPs).  This  concern  and  shrinking  defense  budgets  have 
led  Congress  to  effect  statutory  provisions  that  would  focus  more  attention  of  senior 
policymakers  on  oversight  of  MDAPs1  and  other  large  costly  programs.2  The  Weapon 
Systems  Acquisition  Reform  Act  (WSARA)  of  20093  established  a  number  of  require¬ 
ments  that  affected  the  operation  of  the  Defense  Acquisition  System  and  the  duties  of 
the  key  officials  who  support  it,  including  the  requirement  to  establish  a  new  organiza¬ 
tion  in  the  Office  of  the  Secretary  of  Defense  (OSD)  with  the  mandate  to  conduct  and 
oversee  performance  assessments  and  root  cause  analysis  (PARCA)  for  MDAPs.4  The 
act  assigned  the  PARCA  organization  five  primary  responsibilities: 

1.  Carry  out  performance  assessments  of  MDAPs. 

2.  Perform  root  cause  analysis  (RCA)  of  MDAPs  whose  cost  growth  exceeds  the 
threshold  as  detailed  in  the  Nunn-McCurdy  Act. 

3.  Issue  policies,  procedures,  and  guidance  governing  the  conduct  of  performance 
assessments  and  root  cause  analyses. 

4.  Evaluate  the  utility  of  performance  metrics  used  to  measure  the  cost,  schedule, 
and  performance  of  MDAPs. 


1  U.S.  House  of  Representatives,  House  Report  (HR)  111-124  on  S.  454,  “Weapon  Systems  Acquisition  Reform 
Act  of  2009,”  May  20,  2009;  U.S.  House  of  Representatives,  Committee  on  Armed  Services,  House  Report 
111-101  on  HR  2101,  “Weapons  Acquisition  System  Reform  Through  Enhancing  Technical  Knowledge  and 
Oversight  Act  of  2009,”  May  12,  2009. 

2  Ike  Skelton  Defense  Authorization  Act  for  Fiscal  Year  2011,  December  20,  2010. 

3  Public  Law  111-23,  Weapon  Systems  Acquisition  Reform  Act  of  2009,  May  22,  2009. 

4  Ashton  B.  Carter,  Under  Secretary  of  Defense,  Acquisition,  Technology,  and  Logistics  (OUSD  [AT&L]), 
“Directive-Type  Memorandum  (DTM)  09-027 — Implementation  of  the  Weapon  Systems  Acquisition  Reform 
Act  of  2009,”  December  4,  2009;  Public  Law  111-23,  §  103. 
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5.  Advise  acquisition  officials  on  performance  issues  that  may  arise  regarding  an 
MDAPd 


Purpose 

This  report,  the  second  of  a  continuing  series  of  reports  that  capture  RAND  efforts  to 
support  the  PARCA  office,  has  two  purposes.6  The  first  is  to  provide  additional  root 
cause  analyses  of  two  programs,  the  Army  Excalibur  and  the  Navy  Enterprise  Resource 
Planning  (ERP)  program.  These  two  programs,  although  costly,  are  not  major  plat¬ 
forms,  but  the  analysis  required  differing  approaches.  As  a  result,  researchers  had  dif¬ 
fering  benchmarks  and  metrics  to  contend  with. 

The  second  purpose  is  to  present  an  approach  to  help  identify  the  most  critical 
technical  features  of  a  program.  Critical  program  components  are  the  ones  that  pose 
the  greatest  risk  of  overall  program  failure.  The  approach  is  designed  to  identify  the 
important  program  features  that  decisionmakers  would  want  to  monitor  closely  as  the 
program  progresses.  The  report  uses  an  exercise  to  flag  the  most  critical  features  of  the 
weapon  system,  using  Excalibur  as  an  example.  The  method  and  the  illustration  help 
to  frame  one  approach  for  considering  program  failure  risk  in  structuring  and  review¬ 
ing  programs. 


The  Root  Cause  Analysis  Environment 

Federal  law  shapes  the  environment  in  which  RCAs  are  conducted,  especially  the  time 
available  to  do  them.  In  general,  the  notification  of  an  overrun  and  an  explanation  of 
its  causes  must  occur  quickly.  The  period  of  time  available  for  the  RCA,  while  short  in 
any  case,  can  be  either  45  or  60  days.  10  U.S.  Code  (USC)  §  2433(c)  directs  the  pro¬ 
gram  manager  of  a  major  defense  acquisition  program  to  submit  a  unit  cost  report  to 
the  appropriate  service  acquisition  executive  (SAE)  when  he  or  she  first  determines  that 
there  is  reasonable  cause  to  believe  that  the  program  has  incurred  a  unit  cost  thresh¬ 
old  breach.7  If  the  SAE  makes  the  same  determination  and  the  military  department 
secretary  concurs,  the  secretary  of  the  department  concerned  must  notify  Congress  in 
writing  within  45  days  of  the  program  manager’s  initial  report.8 


6  Public  Law  111-23,  §  103. 

6  See  Blickstein  et  al.,  2011. 

7  For  a  detailed  discussion  of  unit  cost  threshold  breaches,  see  Blickstein  et  ah,  2011. 

8  If  the  program  manager’s  initial  report  is  in  a  quarterly  Selected  Acquisition  Report  (SAR),  then  the  service 
secretary  is  required  to  notify  Congress  in  writing  within  45  days  after  the  end  of  the  quarter.  See  10  USC  §  2433. 
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WSARA  requires  that  a  weapon  system  acquisition  program  undergo  an  RCA 
when  it  incurs  unit  cost  growth  that  exceeds  thresholds  set  by  federal  law.9  WSARA 
directs  the  Secretary  of  Defense  to  initiate  an  RCA  after  consultation  with  the  Joint 
Requirements  Oversight  Council.10  §  103  of  WSARA  assigns  responsibility  for  carry¬ 
ing  out  an  RCA  to  the  senior  official  designated  by  the  Secretary  of  Defense  for  this 
purpose — that  official  is  the  director  of  the  PARCA  office.  10  USC  §  2433(a)  states 
that  notification  to  Congress  of  program  recertification  by  the  Secretary  of  Defense  is 
required  before  the  end  of  the  60-day  period  that  begins  on  the  day  after  the  next  SAR 
is  required  by  10  USC  §  2432(f). 

The  legal  reporting  requirements  described  above  show  that  the  exact  number 
of  days  within  which  an  RCA  must  be  completed  can  vary.  The  variance  stems  from 
two  facts.  The  45-day  period  between  the  program  manager’s  report  and  the  military 
department  secretary’s  notification  to  Congress  of  a  critical  unit  cost  breach  (a  Nunn- 
McCurdy  breach)  starts  the  day  after  the  program  manager  initially  reports  the  breach 
to  the  SAE.* 11  In  contrast,  the  60-day  period  within  which  the  Secretary  of  Defense 
must  submit  a  program  recertification  decision  to  Congress  starts  on  the  day  after  the 
due  date  of  the  first  SAR  that  reports  the  Nunn-McCurdy  breach.  Figures  1.1  and  1.2 
depict  the  process  and  time  lines. 

In  either  case,  not  much  time  is  available  for  the  analysis,  which  has  implications 
for  how  many  data  can  be  collected  and  from  where.  To  illustrate  how  short  a  time 
period  this  might  be,  the  Secretary  of  the  Air  Force  notified  Congress  of  the  Wideband 
Global  Satellite  (WGS)  program  Nunn-McCurdy  breach  on  March  8,  2010.  RAND 
received  word  of  tasking  about  ten  days  later,  and  the  PARCA  office  request  for  initial 
RCA  results  came  during  the  second  week  of  April.  Hence,  the  WGS  RCA  team  had 
approximately  30  calendar  days  to  identify  root  causes  of  this  breach. 

The  discussion  above  shows  that  the  primary  consideration  that  RCA  teams 
should  keep  in  mind  while  designing  a  plan  to  conduct  an  RCA  is  that  it  must  be 
completed  within  a  short  period  of  time.  As  we  will  see  in  the  following  discussion 
on  approaches  to  investigating  common  characteristics  where  an  RCA  methodology 
is  presented,  this  short  time  period  dictates  that  many  tasks  occur  simultaneously  and 
that  a  collaborative  team  effort  is  required  to  integrate  the  specific  findings  into  a  cohe¬ 
sive  narrative  of  events  at  the  root  of  a  Nunn-McCurdy  breach.  Since  analysts  must 
base  their  findings  on  reliable  data,  timely  access  to  relevant  program  information  is 
critical  to  the  success  of  an  RCA.  The  data  aspect  is  discussed  below  and  in  the  data 
sources  section  that  follows.  Finally,  the  RCA  findings  accompany  the  Secretary  of 


9  See  10  USC  §  2432  and  10  USC  §  2433.  Although  the  program  manager,  SAE,  and  military  department 
secretary  must  notify  Congress  of  significant  and  critical  unit  cost  breaches,  the  WSARA  requires  RCAs  only  for 
programs  that  have  incurred  critical  unit  cost  breaches. 

10  See  10  USC  §  2433(a)  of  as  specified  in  §  206  of  Public  Law  111-23. 

11  See  10  USC  §2433. 
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Figure  1.2 

Amount  of  Time  Available  to  Conduct  RCAs  Can  Vary 
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Defense’s  recertification  decision  to  Congress,  so  RCAs  can  play  a  significant  role  in 
determining  the  future  course  of  a  program. 


Candidate  Approaches  to  Investigating  Common  Characteristics 

Characteristics  that  may  accompany  a  Nunn-McCurdy  breach  include  quantity 
changes  and  schedule  delays.  We  discuss  quantity  changes  to  illustrate  approaches  to 
uncovering  the  underlying  factors  that  cause  quantity  changes  to  occur. 

Of  the  six  RCAs  RAND  has  conducted  to  date,  four  incurred  unit  cost  threshold 
breaches  with  associated  quantity  changes.  In  all  four  cases,  however,  quantity  change 
was  not  the  root  cause  of  the  unit  cost  increases.  In  fact,  understanding  the  principle 
that  quantity  change  is  rarely  a  root  cause  for  cost  growth  is  fundamental  to  inves¬ 
tigating  cases  where  quantity  changes  accompany  unit  cost  threshold  breaches.  The 
procurement  quantity  of  every  major  defense  system  acquisition  is  a  carefully  derived 
number  based  on  the  projected  requirements  for  the  system.  The  requirements  analysis 
that  supports  the  quantity  is  a  mandatory  activity  that  occurs  before  any  system  enters 
the  acquisition  life  cycle.  Hence,  a  change  in  quantity  after  a  system  enters  the  acquisi¬ 
tion  life  cycle  occurs  only  when  updated  analysis  shows  an  alternative  that  supports  a 
different  quantity.  If  a  quantity  decrease  occurs,  then  the  research,  development,  test, 
and  evaluation  (RDT&E)  costs  are  spread  over  fewer  units,  which  results  in  a  higher 
program  acquisition  unit  cost  (PAUC) — the  sum  of  development  funding  and  pro¬ 
curement  funding  divided  by  the  number  of  units  procured.12  The  root  cause  of  the 
unit  cost  growth  (higher  PAUC)  flows  not  from  the  quantity  change  but  rather  from 
the  assumptions  and  subsequent  decisions  based  on  the  updated  analysis.  The  RCA 
team  is  charged  with  uncovering  the  basis  of  the  assumptions  and  factors  resulting  in 
quantity  change  decisions  based  on  the  updated  analysis. 

The  RAND  experience  illustrates  why  we  conclude  that  quantity  changes  are  not 
the  source  of  unit  cost  growth.13  For  example,  in  the  case  of  DDG-1000,  the  quan¬ 
tity  was  decreased  from  ten  ships  at  Milestone  (MS)  B  to  three  ships  when  the  PAUC 
breached  the  critical  unit  cost  growth  threshold  and  triggered  the  RCA.  The  DDG- 
1000  RCA  team  found  that  updated  perceived  changes  in  the  emerging  threat  and 
mission  priorities,  as  well  as  affordability,  all  key,  played  roles  in  the  DDG-1000  quan¬ 
tity  decrease  associated  with  the  PAUC  critical  threshold  breach. 

The  RAND  experience  to  date  also  shows  that  although  four  programs  had  asso¬ 
ciated  quantity  changes  when  they  incurred  Nunn-McCurdy  breaches  that  triggered 
RCA  examinations,  in  each  case  the  quantity  change  was  grounded  in  other  program- 


12  Average  procurement  unit  cost  (APUC)  is  the  procurement  funding  divided  by  the  number  of  units  procured. 

13  See  Blickstein  et  ah,  2011,  for  detailed  discussions  of  the  DDG-1000,  Longbow  Apache,  Joint  Strike  Fighter, 
and  WGS  root  cause  analyses. 
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specific  factors  that  resulted  in  unit  cost  growth.14  Uncovering  the  grounds  on  which 
quantity  changes  are  founded  is  an  important  part  of  the  thorough  and  insightful 
RCAs  demanded  by  the  WSARA.  The  RAND  experience  also  points  to  the  impor¬ 
tance  of  understanding  the  program  history,  the  acquisition  environment  throughout 
the  program  life  cycle,  and  the  cost  changes  that  have  occurred  along  the  way.  The 
melding  of  history  with  cost  changes  is  an  important  step  in  an  RCA.  (See  the  meth¬ 
odology  discussion  in  Chapter  Five.) 


Data  Sources 

The  short  period  of  time  in  which  RCAs  must  be  completed  points  to  the  importance 
of  timely  access  to  program  data.  To  uncover  root  causes  of  unit  cost  growth,  the  RCA 
team  must  thoroughly  understand  the  entire  history  of  the  program,  know  what  key 
decisions  were  made  and  why,  comprehend  how  each  significant  cost  change  occurred, 
link  the  cost  history  to  key  program  events,  and  from  this  combined  knowledge  draw 
out  the  salient  events  at  the  root  of  the  Nunn-McCurdy  breach.  Timely  access  to  accu¬ 
rate  and  complete  program  data  is  critical,  especially  in  light  of  the  short  schedule  and 
need  for  several  individuals  to  concentrate  on  a  particular  aspect  of  the  investigation 
while  keeping  abreast  of  the  information  being  generated  by  the  rest  of  the  team. 


Root  Cause  Matrix 

As  discussed  in  our  companion  report,15  RAND  assembled  teams,  one  for  each  system, 
to  respond  to  PARCA’s  request.  Each  team  undertook  two  tasks  in  tandem:  estab¬ 
lishing  the  basic  facts  surrounding  the  program’s  Nunn-McCurdy  breach  or  other 
reported  cost  growth  basis  and  determining  the  contribution  to  unit  cost  growth  of 
the  eight  RCA  issues  stipulated  in  the  WSARA  legislation  and  listed  above.  The  time 
that  elapsed  between  each  program’s  completion  of  Milestone  B  (or  applicable  counter¬ 
part)  and  the  recognition  of  RCA  issues  applicable  to  each  program  were  portrayed  in 
a  chart  similar  to  Table  1.1,  which  illustrates  the  framework  provided  by  the  PARCA 
office.  For  these  programs  under  RAND’s  purview,  this  figure  continues  to  provide  a 
temporal  lens  through  which  RCA  issues  could  be  viewed  and  the  analysis  informed. 


14  Though  not  discussed  in  detail  in  this  report,  schedule  slips  are  another  common  characteristic  that  can 
accompany  unit  cost  breaches.  In  the  case  of  schedule  delays,  an  RCA  team  should  investigate  unforeseen  techni¬ 
cal  issues  and  changes  in  procurement  environment  such  as  labor  disputes,  production  line  problems,  or  lack  of 
timely  availability  of  raw  materials. 

15  See  also  Blickstein  et  al.,  2011. 
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Table  1.1 

PARCA  Root  Cause  Matrix  Framework 


Year  from  MS  B  and  Fiscal  Year 

B 

+1 

+2 

+3 

+4 

+5 

2001 

2002 

2003 

2004 

2005 

2006 

Baseline  issues 

Unrealistic  estimates  for 
cost  or  schedule 

X 

X 

X 

X 

X 

Immature  technology; 
excessive  manufacturing, 
integration  risk 

X 

X 

X 

X 

X 

Unrealistic  performance 
expectations 

X 

X 

X 

X 

X 

Execution  issues 

Changes  in  procurement 

X 

Change  from 

quantity 

150  to  55 

Inadequate  funding/ 
funding  instability 

X 

Unanticipated 
design,  engineering, 
manufacturing,  or 
technical  issues 

X 

Poor  performance  of 
government  or  contract 
personnel 

X 

In  addition  to  the  findings  for  each  program,  a  Navy  ERP-  and  Excalibur-specific  ver¬ 
sion  of  this  chart  can  be  found  in  subsequent  chapters  of  this  report.16 

The  figure  arrays  the  issues  specified  in  the  Nunn-McCurdy  legislation  in  the  left 
column  and  the  fiscal  years  of  a  notional  program  whose  MS  B  (or  other  applicable 
milestone)  occurred  in  2001  across  the  top.  An  “X”  indicates  the  years  in  which  the 
event  occurred. 

As  mentioned  above,  the  time  line  for  reporting  and  certification  restricted  the 
amount  of  data-gathering  and  analysis  that  could  be  performed.  To  meet  PARCA 
needs,  RAND  relied  on  many  documents  through  the  course  of  the  project.  Some  of 
this  material  and  the  program  discussion  process  provided  quantitative  data,  whereas 
some  provided  program  execution  and  management  decisionmaking  insights. 

Analysis  of  each  program  again  identified  risk  as  a  common  denominator  to  all 
programs.  As  the  risks  encountered  by  each  program  were  identified  and  assessed, 
sources  of  program  vulnerabilities  were  collected  and  compared.  Both  sets  of  assigned 
programs  shared  some  vulnerability,  but  others  were  specific  to  only  one. 


16 


See  also  Blickstein  et  al„  2011,  for  root  cause  matrices  for  the  programs  examined  there. 
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Of  note  is  the  fact  that  the  nature  of  the  risk  was  modified  in  the  latest  two  pro¬ 
grams  examined  because  neither  was  a  major  weapon  platform,  as  was  the  case  with 
the  earlier  four  programs  examined.  Further,  one  of  these  two,  Navy  ERP,  does  not 
involve  a  weapon  system,  but  rather  is  a  business  process.  The  other,  Excalibur,  is  a  con¬ 
sumable  that  ties  to  force  structure  considerations  in  a  different  manner  than  do  major 
platforms  and  represents  some  unique  challenges.  Our  analysis  demonstrates  the  point 
that  some  programs  need  to  be  considered  through  a  different  lens  than  do  others.  The 
Excalibur  RCA  illustrates  that  early  and  inaccurate  cost  estimates  can  be  detrimen¬ 
tal  to  a  program’s  success,  but  additional  considerations  have  great  value.  Although  a 
cost-driven  approach  to  an  RCA  is  well  suited  for  major  weapon  programs  such  as  the 
Joint  Strike  Fighter,  additional  methods  are  necessary  for  expendables  such  as  Excali¬ 
bur.  Chapter  Five  introduces  an  approach  to  risk  analysis  of  technical  aspects  using 
the  Excalibur  program  example.  Similar  to  the  cost-oriented  RCA,  the  risk  analysis 
is  based  on  a  performance  time  line  of  select  technical  components.  The  time  line  of 
technical  failures,  delays,  modifications,  and  other  challenges  depicts  a  longer  history 
of  risk  than  several  high-level  reports  would  suggest. 

Organization  of  the  Report 

This  report  contains  six  chapters.  Chapters  Two  and  Three  report  the  findings  of  the 
RCAs  performed  by  the  RAND  teams  on  each  of  the  last  two  programs  under  review: 
Excalibur  and  Navy  ERR  These  chapters,  idiosyncratic  of  the  specific  programs  and 
independent  analyses  of  the  teams,  reflect  the  reports  that  were  sent  to  the  PARCA  office 
to  be  used  to  carry  out  its  responsibilities  and  produce  both  the  materials  necessary  for 
the  recertification  decision  process  as  required  by  OUSD  (AT&L),  and,  ultimately,  the 
Secretary  of  Defense,  as  well  as  memoranda  to  management  addressing  issues  of  con¬ 
cern  for  consideration  as  the  congressional  appetite  for  oversight  and  reporting  contin¬ 
ues  to  evolve.  Chapter  Four  details  an  approach  designed  by  the  authors  to  help  identify 
the  most  critical  features  of  a  program.  These  critical  program  components  are  those 
that  carry  the  most  risk  of  overall  program  failure.  The  approach  is  designed  to  identify 
the  important  program  features  that  decisionmakers  would  want  to  concentrate  on  as 
the  program  develops  over  time.  With  Excalibur  as  the  example,  Chapter  Five  uses 
this  selective  screening  of  critical  components  exercise  to  initially  flag  the  most  criti¬ 
cal  features  of  the  weapon  system.  This  chapter  outlines  a  process  for  identifying  the 
level  of  detail  particularly  appropriate  for  a  specific  inquiry  effort  to  uproot  program 
risk.  Whereas  the  process  discussed  in  Chapter  Four  is  designed  to  identify  the  critical 
components  of  a  program  that  likely  contain  the  most  risk,  Chapter  Five  outlines  the 
type  of  detailed  review  into  those  components  that  may  be  necessary  given  the  initiat¬ 
ing  hypotheses.  Together,  Chapters  Four  and  Five  help  to  frame  one  approach  for  con¬ 
sidering  risk  of  program  failure  in  programs  that  have  not  yet  breached.  Chapter  Six 
presents  our  concluding  observations. 


CHAPTER  TWO 


Excalibur 


This  chapter  describes  the  Nunn-McCurdy  breach  in  the  Army’s  Excalibur  program 
(XM982  155mm  extended-range  guided  artillery  projectile)  encountered  in  August 
2010.  The  chapter  begins  with  an  overview  of  the  Excalibur  program.  It  then  describes 
the  general  circumstances  of  the  Nunn-McCurdy  breach  and  follows  that  with  a  more 
detailed  cost  history  of  the  program,  including  a  discussion  of  RDT&E  and  procure¬ 
ment  cost  histories  as  well  as  total  program  costs.  The  chapter  concludes  with  findings 
and  some  considerations  for  the  future. 

This  analysis  of  Excalibur’s  Nunn-McCurdy  experience  is  not  intended  to  be  a 
complete  program  history  and  so  does  not  attempt  to  deal  with  every  element  of  the 
program.  Rather,  we  have  attempted  to  identify  those  aspects  of  the  program  that  are 
relevant  to  the  explanation  of  the  Nunn-McCurdy  unit  cost  breaches.  In  this  case,  our 
analysis  focuses  on  reasons  for  quantity  changes  throughout  the  program,  technical 
challenges,  and  other  factors  that  may  account  for  the  cost  growth  experienced  by  the 
program. 


Program  Overview 

Excalibur  is  an  Acquisition  Category  (ACAT)  IC  Army  munition  program  that  fulfills 
the  Army’s  requirement  for  precision  fires  in  an  artillery  munition.  The  Army  wanted 
an  artillery  round  compatible  with  both  existing  systems  (M777A2  lightweight  155mm 
howitzer  and  the  M109A6  155mm  Paladin  howitzer)  and  planned  future  systems  that 
increased  range  and  accuracy  while  decreasing  collateral  damage.  The  Excalibur  prod¬ 
uct  manager  reports  through  the  combat  ammunition  systems  program  manager,  who 
reports  to  the  Program  Executive  Officer  for  Ammunition  (PEO-AMMO).  The  Mile¬ 
stone  Decision  Authority  for  Excalibur  is  the  Army  Acquisition  Executive  (AAE). 

Excalibur  is  a  complicated  acquisition  program  whose  history  traces  back  to  the 
early  1990s.  That  history  has  not  been  untroubled.  In  the  past  nearly  20  years,  Excali¬ 
bur  has  gone  through  major  technical  modifications,  multiple  major  decreases  in  the 
planned  quantity  of  projectiles,  two  related  program  cancellations  (Crusader  and 
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NLOS-C),  and  an  acceleration  in  production  initiation  to  fulfill  an  urgent  require¬ 
ment  in  Operation  Iraqi  Freedom  (OIF). 

Excalibur’s  program  history  began  in  1992  with  an  MS  0  decision  in  1992.  In 
1995,  the  program  was  an  advanced  development  program  with  an  entirely  differ¬ 
ent  focus  from  today’s.  Excalibur  evolved  from  an  unguided  munition  with  increased 
range  (1997)  to  a  full  global  positioning  system  (GPS)  and  inertial  measurement  unit 
(IMU)  guidance  kit  (to  2010).  In  May  1997,  as  an  ACAT  III  program,  the  initial 
Excalibur  Operational  Requirements  Document  (ORD)  was  approved  along  with  an 
MS  I/II  combined  decision.  Initial  program  quantity  was  200,000,  and  Texas  Instru¬ 
ments  (since  acquired  by  Raytheon)  won  a  competitive  selection  to  serve  as  the  system 
contractor.  The  initial  engineering  manufacturing  and  development  contract  with 
Raytheon  TI  Systems,  Inc.,  was  awarded  on  January  23,  1998. 

In  May  2001,  the  program’s  ACAT  designation  changed  from  ACAT  III  to 
ACAT  II.  In  November  of  that  year,  the  AAE  directed  that  the  MS  I/II  decision  be 
accepted  as  the  official  MS  B  decision.  Six  months  later,  the  program  became  an  ACAT 
ID  program  with  quantity  set  at  76,677,  down  considerably  from  the  initial  quantity 
established  in  1997.  In  addition,  because  of  the  cancellation  of  the  Crusader  program, 
Excalibur  was  restructured  to  include  the  Future  Combat  System’s  non-line-of-sight 
(NLOS)  cannon.  Both  the  quantity  and  ACAT  status  were  changed  shortly  thereafter 
to  a  quantity  of  61,483  in  2003,  a  20  percent  decrease,  and  an  ACAT  status  of  1C. 

In  March  2004,  the  Excalibur  program  merged  with  a  joint  Swedish/U.S.  pro¬ 
gram  known  as  the  “Trajectory  Correctable  Munitions.”1  This  partnership  had  a  posi¬ 
tive  effect  on  the  program  because  it  enabled  the  program  to  overcome  some  design 
hurdles.  A  revised  ORD  was  also  approved  in  September  2004.  This  revised  ORD 
reflected  a  major  design  change  with  the  deletion  of  the  “Dual  Purpose  Improved 
Conventional  Munitions)  variant.”  This  ORD  also  included  the  addition  of  the  dis¬ 
criminating  munitions  variant  and  a  three-increment  approach  that  remains  current: 

•  Increment  la- 1  projectile:  available  for  early  fielding;  met  requirements  for  lethality 
and  accuracy  in  a  nonjammed  environment. 

•  Increment  la-2  projectile :  designed  to  meet  requirements  for  accuracy  in  a  jammed 
environment,  with  extended  range  and  increased  reliability. 

•  Increment  lb  projectile:  improved  reliability;  lowered  unit  costs;  and  could  be 
fielded  in  fiscal  year  (FY)  14. 2 


1  Excalibur  has  an  agreement  with  Sweden  in  which  Sweden  contributes  development  resources.  Also,  sev¬ 
eral  countries  have  bought  projectiles,  including  Australia,  Canada,  and  the  United  Kingdom.  In  addition,  the 
Marine  Corps  buys  projectiles  for  use  in  OIF  and  Operating  Enduring  Freedom  (OEF)  from  the  Excalibur  pro¬ 
gram  manager. 

2  Raytheon  Missile  Systems  developed  Increments  la-1,  la-2,  and  lb.  Alliant  Techsystems,  Inc.,  developed  only 
Increment  lb.  Raytheon  produces  or  will  produce  all  increments. 
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Subsequently,  in  October  2004,  an  acquisition  program  baseline  (APB)  was 
approved  that  reflected  a  major  reduction  in  the  number  of  projectiles.  A  capabilities- 
based  analysis  conducted  in  the  work-up  to  new  baseline  determined  that  a  procure¬ 
ment  quantity  of  61,483  rounds  was  needed.  However,  the  AAE  approved  the  October 
2004  APB  with  an  Army  procurement  objective  of  30,000  rounds  as  supported  by  the 
Army  Cost  Position.  This  reduction  suggests  that  affordability  concerns  (i.e.,  total  pro¬ 
gram  cost)  were  the  dominant  determinant  of  the  baseline  quantity  objective. 

In  March  2005,  the  Army  Resources  and  Requirements  Board  validated  an  urgent 
needs  statement  for  a  precision  artillery  round,  submitted  by  the  Combined  Forces 
Land  Component  Command.3  As  a  result,  the  program  accelerated  the  fielding  and 
testing  of  its  Increment  la-1.  An  MS  C  Acquisition  Decision  Memorandum  (ADM) 
was  signed  in  May  2005  with  a  low  rate  initial  production  approval  of  500  Block  la-1 
projectiles,  and  a  production  contract  was  awarded  in  June  2005  to  Raytheon  for  165 
Excalibur  Block  la-1  projectiles.  However,  Raytheon  experienced  some  quality  issues 
that  delayed  production  and  Increment  la-2  qualification. 

These  issues  also  had  some  negative  effects  on  the  program’s  cost  and  schedule.4 
Throughout  the  remainder  of  2005,  several  tests  were  conducted  on  Increment  la-1, 
and  soldier  training  was  initiated.  Testing  of  Block  la-1  and  development  of  Block  la-2 
continued  throughout  2006.  After  a  successful  limited  user  test  in  February  2007, 
Increment  la-2  also  passed  MS  C  in  July  2007. 

In  September  2008,  the  Joint  Munitions  and  Lethality  Center  awarded  two  con¬ 
tracts  (to  Raytheon  and  Affiant  Techsystems,  Inc.)  for  the  Increment  lb  demonstration 
phase.  At  the  completion  of  the  first  phase,  both  contractors  were  required  to  partici¬ 
pate  in  a  shoot-off  and  competitive  down-selection. 

The  program  office  fired  seven  Excalibur  la-1  projectiles  at  Yuma  Proving  Ground 
in  March  2009.  This  test  confirmed  that  the  Honeywell  IMU  did  not  meet  the  la-1 
performance  requirement,  so  the  program  transitioned  to  the  Atlantic  Inertial  Systems 
(AIS)  IMU.  While  the  problem  was  being  rectified,  no  deliveries  were  made  from 
November  2008  through  August  2009.  Overcoming  this  technical  obstacle  was  a  sig¬ 
nificant  achievement  for  Excalibur.  In  December  2009,  481  rounds  of  Excalibur  were 
shipped  to  theater.  Two  months  later  (February  2010),  Excalibur  completed  its  initial 
operational  test  and  evaluation  (IOT&E). 

In  April  2010,  the  Vice  Chief  of  Staff  of  the  Army  Precision  Fires  Capability 
Portfolio  Review  decision  reduced  quantities  further,  from  30,000  to  6,264,  a  deci¬ 
sion  that  the  Configuration  Steering  Board  also  supported.  This  decision  resulted  in  a 
critical  Nunn-McCurdy  unit  cost  breach,  and  a  Program  Deviation  Report  was  sub¬ 
mitted  by  the  Excalibur  program  manager  in  July  2010,  and  notification  to  Congress 


3  Excalibur  Acquisition  Strategy  Report,  April  2007,  p.  6. 

4  Government  Accountability  Office,  Defense  Acquisitions:  Assessments  of  Selected  Weapons  Programs,  Washing- 
ton,  D.C.:  U.S.  Government  Printing  Office,  GAO-10-388SP,  March  2010,  pp.  65—66. 


14  Root  Cause  Analyses  of  Nunn-McCurdy  Breaches,  Volume  2 


of  the  breach  followed  in  the  next  month.  Full  rate  production  and  initial  operational 
capability  (IOC) — both  scheduled  for  2010 — were  put  on  hold  until  after  the  Nunn- 
McCurdy  recertification  process.  However,  the  recently  completed  competition  to  pro¬ 
duce  the  Increment  lb  rounds  did  result  in  a  source  selection  decision  in  late  September 
2010:  Raytheon  was  selected  to  produce  the  Increment  lb  rounds. 


Research  Approach 

The  information  used  in  this  analysis  was  drawn  from  official  primary  source  doc¬ 
umentation.  We  reviewed  a  wide  range  of  documentary  evidence  including  ADMs, 
acquisition  strategies,  APBs,  Defense  Acquisition  Executive  Summary  (DAES),  SARs, 
Army  budget  material,  and  cumulative  earned  value  management  system  data  on  the 
major  Excalibur  contracts.  Other  key  information  sources  included  a  briefing  given  to 
the  Nunn-McCurdy  Integrated  Product  Teams,  Excalibur’s  Program  Deviation  Report 
(PDR),  and  the  Army’s  Munitions  Mix  Study.  In  addition,  we  interviewed  program 
office  personnel  and  OSD  officials.  Finally,  we  conducted  a  thorough  search  of  the 
trade  literature  and  Government  Accountability  Office  (GAO)  audits  of  the  program. 
Sources  used  in  this  RCA  appear  in  the  list  of  references  at  the  end  of  the  report. 


Cost  History 

This  section  discusses  the  cost  history  of  the  Excalibur  program,  beginning  with  the 
unit  cost  breaches  that  triggered  the  need  for  this  root  cause  analysis.  We  also  examine 
how  RDT&E  and  procurement  costs  have  changed  over  time  and,  to  the  extent  pos¬ 
sible,  identify  factors  affecting  those  changes. 

The  Nunn-McCurdy  Breaches 

On  August  20,  2010,  the  Secretary  of  the  Army  officially  reported  to  Congress  that  the 
Excalibur  program  had  experienced  unit  cost  growth  exceeding  the  critical  statutory 
unit  cost  growth  thresholds.  A  reduction  in  the  planned  total  quantity  from  30,000 
to  6,264  projectiles  resulted  in  these  unit  cost  breaches,  and  the  Excalibur  program 
entered  the  Nunn-McCurdy  recertification  process.  This  process  was  updated  by  Con¬ 
gress  in  the  WSARA  of  2009  to  include  a  root  cause  analysis  conducted  by  the  newly 
established  PARCA  office. 

In  accordance  with  10  USC  §  2433,  the  Army  notified  Congress  that  based  on  a 
July  6,  2010,  Program  Deviation  Report,  an  approximately  890  percent  reduction  in 
quantity  resulted  in  Excalibur  unit  cost  growth  exceeding  the  critical  statutory  acquisi¬ 
tion  program  baselines. 

The  Excalibur  program  incurred  four  critical  Nunn-McCurdy  breaches: 
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•  The  APUC  exceeded  the  current  2007  APB  by  115.91  percent. 

•  The  PAUC  exceeded  the  2007  APB  by  181.34  percent. 

•  The  APUC  exceeded  the  original  2004  APB  by  143.59  percent. 

•  The  PAUC  exceeded  the  original  2004  APB  by  193.06  percent.5 

The  2009  WSARA  requires  a  root  cause  analysis  process  on  programs  that  incur 
critical  breaches.6 7  The  “speeding  ticket”  addressed  by  the  Excalibur  root  cause  analysis 
is  shown  in  Table  2.17 

Research,  Development,  Test,  and  Evaluation  Cost  Profiles 

The  Excalibur  RDT&E  funding  line  included  development  of  all  Excalibur  variants 
and  technology  upgrades.  In  addition,  a  number  of  other  efforts  share  the  Excali¬ 
bur  RDT&E  funding  line.  These  include  the  Spin  Stabilized  Sensor  Fused  Munition 
(SSSFM);  nonlethal  munitions;  a  program  to  evaluate  smart  submunitions  for  poten¬ 
tial  cannon,  missile,  and  rocket  applications;  and  the  105mm  cargo  projectiles  that 
were  added  by  Congress  in  FY2008-FY2009.  Despite  the  large  scope  of  RDT&E 
included,  the  congressional  addition  of  the  105  projectile,  and  the  technologically 
significant  development  of  the  Excalibur  variants,  RDT&E  funding  has  been  fairly 
stable.  A  notable  exception  is  the  extension  of  RDT&E  funding  beyond  2008  starting 
with  the  December  2003  RDT&E  funding  profile. 

Figure  2.1  shows  the  RDT&E  funding  profiles  from  all  of  the  Excalibur  SARs 
from  December  2002  through  December  2009  and  the  August  2010  DAES.8  Other 
than  the  extension  of  funding  to  years  past  2008,  the  profiles  are  all  close  to  each 
other,  indicating  stable  annual  RDT&E  funding.  The  chart  also  shows  RDT&E 
quantities  associated  with  some  profiles.  Though  RDT&E  quantities  increased  several 
times,  there  was  little  notable  RDT&E  cost  growth  as  a  result.  In  general,  the  planned 
RDT&E  expenditures  for  a  given  year  were  spent  according  to  plan. 


5  For  the  purposes  of  this  report,  the  current  estimate  cost  data  are  taken  from  a  DAES/Web  Services  current 
status  report  downloaded  from  Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  on  August 
24,  2010.  An  Excalibur  September  2010  draft  SAR  posted  on  DAMIR  in  October  contains  slightly  different 
current  estimate  cost  and  quantity  data.  The  September  2010  draft  SAR  shows  that  the  APUC  exceeded  the 
2004  and  2007  baselines  by  159  percent  and  130  percent,  respectively,  and  the  PAUC  exceeded  the  2004  and 
2007  baselines  by  211  percent  and  199  percent,  respectively.  Total  procurement  quantity  is  shown  as  6,506  vs. 
the  6,905  shown  in  the  August  2010  DAES.  In  both  documents,  the  RDT&E  quantity  is  shown  as  544  rounds. 
Although  these  cost  growth  numbers  are  higher  than  those  reported  in  the  August  DAES,  the  more  recent  cur¬ 
rent  estimate  does  not  change  the  analysis  or  findings  of  factors  affecting  unit  cost  growth  presented  here. 

6  Public  Law  111-23,  May  22,  2009. 

7  Speeding  ticket  is  the  term  used  by  the  PARCA  office  to  describe  the  event  that  triggers  the  need  for  an  RCA. 

8  The  August  24,  2010,  DAES  contains  the  latest  DAES  information  available  as  of  this  writing. 


Table  2.1 

Excalibur  "Speeding  Ticket" 


Current 

Estimate  (August 
2010  SAR) 

FY  2007 
$  thousands) 

Cost  Growth  Threshold  Breaches 

Program 

Baseline 

Unit  Cost 
(FY  2007 
$  thousands) 

Baseline 

Breached 

Percentage 

Amount  of 
Unit  Cost 
Change 

Level 

Baseline 

Quantity 

August  2010 
DAES 

Immediate 
Cause  in 
August 

2010  DAES 

Excalibur 

APUC 
$44.40 
(2007  APB) 

APUC 

$94.76 

Over  current 

baseline 
(2007  APB) 

APUC 

+115.91% 

+$51 K 

FY  2007  $K 

Critical 

30,000 

6,905 

Quantity 
decreased  to 
6,905  units 

PAUC 
$74.52 
(2007  APB) 

PAUC 

$211.32 

PAUC 

+181.34% 

+$136K 

FY  2007  $K 

Critical 

30,388 

7,449 

APUC 
$39.43 
(2004  APB) 

APUC 

$94.76 

Over  original 
baseline 
(2004  APB) 

APUC 

+143.59% 

+$56K 

FY  2007  $K 

Critical 

30,000 

6,905 

PAUC 
$71.61 
(2004  APB) 

PAUC 

$211.32 

PAUC 

+193.06% 

+$139K 

FY  2007  $K 

Critical 

30,269 

7,449 

SOURCES:  Department  of  Defense,  Defense  Acquisition  Executive  Summary,  August  24,  2010;  Department  of  Defense,  Defense  Acquisition 
Executive  Summary,  July  29,  2010;  Under  Secretary  of  the  Army,  Excalibur  Program  Acquisition  Decision  Memorandum,  May  12,  2010; 
Department  of  Defense,  Selected  Acquisition  Report  Excalibur,  December  31,  2009;  Department  of  Defense,  Selected  Acquisition  Report 
Excalibur,  December  31,  2007;  Department  of  Defense,  Selected  Acquisition  Report  Excalibur,  September  30,  2007;  Department  of  Defense, 
Selected  Acquisition  Report  Excalibur,  December  31,  2006;  Department  of  Defense,  Selected  Acquisition  Report  Excalibur,  December  31,  2005; 
Department  of  Defense,  Selected  Acquisition  Report  Excalibur,  December  31,  2004;  Department  of  Defense,  Selected  Acquisition  Report 
Excalibur,  December  31,  2003;  Department  of  Defense,  Selected  Acquisition  Report  Excalibur,  December  31,  2002. 

NOTE:  The  numbers  in  red  indicate  the  "speeding  ticket"  triggering  root  cause  analysis  by  PARCA. 
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Figure  2.1 

Excalibur  RDT&E  Funding  Profile 


Fiscal  year 

SOURCES:  Derived  from  Excalibur  SARs,  December  2002  to  December  2009;  Excalibur  DAES,  August  24, 
2010  (downloaded  from  DAMIR). 

NOTE:  BY  is  base  year. 
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Procurement  Cost  Profiles 

Annual  Excalibur  procurement  funding  was  also  fairly  stable  from  2004  to  2009  (see 
Figure  2.2).  The  December  2002  and  December  2003  SARs  show  marked  changes 
in  the  procurement  funding  profiles  after  2007.  These  differences  likely  result  from 
the  reduction  in  procurement  quantity  from  76,408  units  in  2002  to  61,483  units  in 
2003,  and  then  to  30,000  rounds  in  2004. 9  The  Army  procurement  quantity  objec¬ 
tive  remained  constant  at  30,000  units  from  2004  through  2009,  and  the  procure¬ 
ment  funding  profile  did  not  change  significantly  during  these  years.  The  August  2010 
DAES  reflects  the  procurement  reduction  to  6,905  units,  and  the  procurement  fund¬ 
ing  profile  in  the  August  2010  DAES  shows  that  the  change  is  reflected  in  a  truncation 
of  procurement  funding  after  2014. 


9  The  total  procurement  quantity  is  actually  6,905  and  is  composed  of  6,264  production  rounds,  242  rounds 
fired  during  production  acceptance  testing,  and  399  rounds  of  foreign  military  sales  buy-back.  From  a  unit  cost 
point  of  view,  6,905  rounds  is  the  relevant  total  quantity  number. 
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Figure  2.2 

Excalibur  Procurement  Funding  Profiles 


.  afrom  December  2004  to 

Fiscal  year  December  2009  =  30,000 

SOURCES:  Derived  from  Excalibur  SARs,  December  2002  to  December  2009;  Excalibur  DAES,  August  24, 
2010  (downloaded  from  DAMIR). 
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The  increase  in  Increment  la-1  and  la-2  rounds  and  associated  production  accel¬ 
eration  in  support  of  the  urgent  operational  need  in  OIF/OEF  is  not  noticeable  in  the 
annual  funding  profiles.10 

Total  Program  Funding  and  Quantity  Profiles 

As  might  be  expected,  the  total  Excalibur  funding  profile  is  shaped  by  the  procurement 
profile.  Figure  2.3  shows  a  synthesis  of  the  RDT&E  funding,  procurement  funding, 
and  total  RDT&E  and  procurement  quantity  profiles.  After  the  large  decrease  in  pro¬ 
curement  funding  from  2003  to  2004,  the  ratio  of  procurement  to  RDT&E  fund¬ 
ing  stayed  roughly  the  same  through  2009,  with  procurement  accounting  for  57-61 
percent  of  total  program  costs.  With  the  latest  reduction  in  quantity,  this  ratio  has 
reversed;  RDT&E  now  accounts  for  58  percent  of  total  Excalibur  program  costs  versus 
the  previously  roughly  40  percent. 


10  The  2006  and  2009  SARs  identify  a  small  amount  of  funding  that  appears  to  support  the  urgent  need  request: 
$14.1  million  and  $11.7  million,  respectively  (BY  2007  dollars).  We  have  not  been  able  to  find  documentary  evi¬ 
dence  of  additional  funds  beyond  this. 
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Figure  2.3 

Excalibur  Total  Program  Funding  and  Quantity  Profiles 
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Unit  Cost  Profile 

The  SAR  data  show  that  both  the  APUC  and  PAUC  remained  fairly  stable  from 
December  2002  through  December  2009  (Figure  2.4).  Of  note  is  the  slight  increase  in 
the  PAUC  from  2002  to  2003,  whereas  the  APUC  decreased  slightly  during  the  same 
period.  The  difference  between  the  PAUC  and  APUC  then  remained  fairly  constant 
until  August  2010  when  the  Army  formally  declared  unit  cost  growth  breaches.  The 
large  reduction  in  quantity  first  reported  in  the  August  2010  DAES  is  reflected  in  the 
large  increases  in  APUC  and  PAUC. 

Figure  2.4  shows  the  APUC,  PAUC,  and  quantity  profiles.  Both  cost  metrics 
should  be  sensitive  to  changes  in  procurement  quantity.  Nevertheless,  the  very  large 
quantity  reductions  in  2003  and  2004  did  not  result  in  any  significant  change  to  either 
APUC  or  PAUC.  This  suggests  that  the  early  unit  cost  estimates  were  not  based  on 
a  complete  analysis.  The  procurement  funding  change  from  2002  to  2004  represents 
a  67  percent  reduction  from  the  2002  amount,  but  neither  unit  cost  metric  changed 
much  at  all.  This  insensitivity  also  calls  into  question  the  realism  of  the  unit  cost  goals 
through  2009.  To  date,  we  have  uncovered  no  additional  funding  sources  that  may 
have  been  used  to  cover  costs  but  are  not  included  in  the  sources  we  examined  for  this 
analysis. 
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Figure  2.4 

Excalibur  Unit  Cost  Profiles 
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SOURCES:  Derived  from  Excalibur  SARs,  December  2002  to  December  2009;  Excalibur  DAES,  August  24, 
2010  (downloaded  from  DAMIR). 
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RDT&E  History 

Figure  2.5  shows  the  total  program  estimated  RDT&E  costs  for  Excalibur  since  2002. 
The  large  increase  from  2002  to  2003  was  due  to  “the  addition  of  Block  lb  of  the  spiral 
development  previously  unfunded  (Engineering).”11  In  other  words,  Block  lb  develop¬ 
ment  had  been  planned,  but  it  was  not  funded  in  the  program  budget  until  2003.  After 
that  increase,  the  RDT&E  budget  appears  to  have  remained  largely  stable.  There  were 
multiple  minor  inflation  and  budget  adjustments  as  well  as  increases  in  the  number  of 
test  rounds,  but  these  changes  were  relatively  small  and  effectively  canceled  each  other 
out.  The  exception  is  the  additional  development  test  quantity  change  from  388  to  544 
to  “account  for  additional  planned  system-level  Increment  lb  test  rounds.”12  The  reason 
for  the  $23.8  million  (BY  2007  dollars)  increase  from  2009  to  2010  is  unknown.13 


11  Excalibur  December  2003  SAR  cost  variance  table  current  change  explanations. 

12  Excalibur  December  2009  SAR  cost  variance  table. 

13  It  is  possible  that  this  increase  is  driven  by  changes  in  the  buy  profile  for  the  RDT&E  rounds,  but  we  do  not 
have  documentary  evidence.  The  August  24,  2010,  DAES  (from  DAMIR)  shows  a  buy  profile  for  development 
rounds  but  previous  SARs  do  not. 


Excalibur  21 


Figure  2.5 

Excalibur  Total  Program  RDT&E  History 
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SOURCES:  Derived  from  Excalibur  SARs,  December  2002  to  December  2009;  Excalibur  DAES,  August  24, 
2010  (downloaded  from  DAMIR). 
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Procurement  History 

Figure  2.6  shows  the  procurement  history  for  Excalibur  from  2002  to  2010.  The  three 
very  large  decreases  in  total  quantity  clearly  drive  the  cost  increase.  The  first  two  quan¬ 
tity  decreases  resulted  in  a  68  percent  reduction  in  procurement  costs  from  what  was 
planned  in  2002.  As  mentioned  above,  it  is  interesting  that  such  a  significant  reduc¬ 
tion  in  quantity  appears  not  to  have  affected  unit  cost.  The  last  quantity  decrease — the 
nearly  80  percent  decrease  that  resulted  in  the  Nunn-McCurdy  unit  cost  breaches — 
reduced  total  procurement  costs  by  52  percent  of  what  had  been  planned  in  2009.  It 
also  resulted  in  a  situation  in  which  the  total  program  RDT&E  funding  ($919.8  mil¬ 
lion,  BY  2007  dollars)  is  about  one-third  higher  than  the  total  procurement  funding 
($654.3  million,  BY  2007  dollars). 

Figure  2.6  also  shows  the  relatively  minor  growth  (17  percent)  in  total  estimated 
procurement  cost  that  occurred  over  the  period  2004  to  2009.  Procurement  quantity 
was  fixed  over  this  period  at  30,000.  Factors  other  than  quantity  accounting  for  this 
17  percent  increase  include  the  following: 

•  additional  funding  to  address  an  urgent  need  for  early  fielding 

•  multiple  small  inflation  adjustments  over  the  period 

•  changes  in  production  profile  (number  procured  each  year,  production  stretchout). 
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Figure  2.6 

Excalibur  Total  Program  Procurement  History,  2002-2010 
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The  largest  increase  occurred  in  2007  (as  reported  in  the  September  2007  SAR): 
$141.5  million  (2007  dollars)  associated  with  the  approved  production  baseline  (July 
2007  APB).  Specifically,  the  September  2007  SAR  states  that  the  program  “received 
additional  program  funding  primarily  in  the  extended  planning  period  to  align  with 
the  agreed  Army  Cost  Position  and  APB  agreement  (Estimating).”14 


Findings15 

Excalibur  formally  began  in  May  1997  as  an  ACAT  III  program  with  approval  of  Mile¬ 
stone  I/II,  which  included  approval  of  an  ORD  and  other  documentation  required  of 
an  ACAT  III  program.  The  program  was  structured  as  an  incremental  development 
in  three  blocks,  defined  by  the  characteristics  of  the  warhead:  Block  I  was  a  unitary 
warhead,  Block  II  was  a  smart  warhead,  and  Block  III  was  a  discriminating  warhead. 

The  original  Excalibur  concept  differed  significantly  from  the  current  one.  The 
original  concept  focused  on  increasing  range  by  means  of  a  rocket-assisted  base  con- 


14  See  the  cost  variance  table  in  the  Excalibur  September  2007  SAR.  A  large  procurement  variance  was  reported 
in  the  December  2004  SAR:  $164.5  million  (then-year  dollars)  associated  with  a  “stretchout  of  the  annual  pro¬ 
curement  buy  profile  (Schedule).”  According  to  the  December  2004  SAR,  the  cost  impact  was  recorded  only  in 
then-year  dollars,  with  no  impact  to  the  base  year  dollar  estimate. 

15  All  dollar  figures  in  this  section’s  narrative  are  expressed  in  base  year  2007  dollars. 
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figuration,  rather  than  on  increasing  accuracy.  It  was  “self-locating,”  with  GPS  and 
transceivers  planned  for  only  one-sixth  of  the  total  number  of  rounds.  However,  the 
initial  contract  to  Texas  Instruments  (later  purchased  by  Raytheon)  in  January  1998 
reflected  a  significantly  different  concept  and  included  GPS,  an  IMU,  and  fin-and- 
canard  gliding  airframe  technology.16  The  initial  low  cost  estimate  and  high-quantity 
target  appear  to  have  resulted  from  a  presumption  that  the  system  did  not  differ  much 
from  older  nonprecision  artillery  rounds.17  However,  the  two  later  baselines  (APBs 
approved  in  October  2004  and  July  2007)  appear  to  contain  cost  and  schedule  esti¬ 
mates  more  consistent  with  the  capabilities  of  the  current  Excalibur  system. 

Root  Cause  Analysis 

Early  program  cost  estimates  were  highly  inaccurate,  as  demonstrated  by  the  insensi¬ 
tivity  of  unit  cost  estimates  to  significant  reductions  in  quantity  in  the  early  program. 

Quantity  reductions  before  200218  and  in  2003  and  2004  appear  to  have  been 
driven  by  affordability  concerns.  Until  the  recent  Munitions  Mix  Study  (formally,  the 
Precision  Munitions  Resourcing  Strategy  Study  conducted  by  the  Center  for  Army 
Analyses),  the  only  capabilities-based  requirements  analysis  mentioned  by  program 
officials  or  in  program  documentation  was  one  in  2004  that  recommended  a  quantity 
of  61, 483. 19  This  requirements  analysis  was  performed  as  part  of  the  activities  leading 
to  Excalibur’s  first  ACAT  I  baseline,  eventually  approved  in  October  2004.  However, 
that  baseline  (the  development  baseline)  included  an  Army  procurement  objective  of 
30,000  rounds.  The  procurement  quantity  objective  of  30,000  remained  the  official 
objective  until  the  May  2010  ADM  reduced  the  quantity  to  6,264. 

According  to  cost  variance  reported  in  the  Excalibur  SARs  from  2002  to  2009, 
the  program  experienced  relatively  minor  cost  growth  in  both  RDT&E  and  procure¬ 
ment  accounts.  As  reported  in  the  December  2009  SAR,  the  APUC  grew  to  20.5 
percent  above  the  October  2004  APB  (from  $39,000  to  $47,000  in  BY  2007  dollars) 
and  6.8  percent  above  the  July  2007  APB  (from  $44,000  to  $47,000  in  BY  2007  dol¬ 
lars).  Factors  affecting  cost  growth  during  this  period  included  budget  adjustments, 
programming  for  unfunded  development  rounds,  and  additional  funding  to  support 
a  validated  urgent  operational  need  (which  included  accelerating  the  production  of 
Increment  la-1  rounds  and  procuring  more  of  those  rounds  than  originally  planned). 
There  were  also  some  technical  difficulties  experienced  during  developmental  testing, 


16  The  addition  of  the  GPS  and  IMU  added  important  technical  complexity  to  the  project.  An  early  risk  analysis 
could  have  raised  warning  flags  about  the  project’s  likelihood  of  success. 

17  Excalibur  Acquisition  Strategy  Report,  April  2007. 

18  According  to  discussions  with  program  officials  and  GAO  MDAP  reports,  quantity  estimates  before  2002 
range  from  over  200,000  rounds  to  76,000  rounds. 

19  This  quantity  (61,483  rounds)  appears  as  the  current  estimate  in  the  Excalibur  December  2003  SAR. 
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production  process  challenges,  relocation  of  contractor  facilities,  and  replacement  of 
the  IMU  vendor. 

Because  of  the  abbreviated  amount  of  time  available  to  conduct  an  RCA,  these 
cost  growth  factors  were  not  initially  evaluated.  However,  RAND  subsequently  devel¬ 
oped  a  complementary  methodology  for  considering  them  as  well  as  a  way  to  under¬ 
stand  the  nature  of  technical  risk.  Chapter  Four  of  this  report  describes  the  comple¬ 
mentary  methodology,  and  Chapter  Five  characterizes  technical  risk  and  how  it  may 
be  evaluated  better. 

None  of  these  factors  appears  to  have  influenced  the  May  2010  decision  to 
reduce  the  Army’s  procurement  objective  from  30,000  to  6,264,  triggering  the  Nunn- 
McCurdy  breach.  Rather,  that  quantity  reduction  decision  appears  to  be  the  result  of 
several  other  factors,  including  the  Precision  Fires  Capability  Portfolio  Review  (also 
called  the  Munitions  Mix  Study),  the  Quantitative  War  Reserve  Requirements  Muni¬ 
tions  Model  Process,  and  reportedly  low  use  of  the  Increment  la-1  round  in  theater.20 

These  three  influential  factors  suggest  a  combination  of  requirements  change  and 
affordability  considerations  as  the  drivers  of  the  decision  to  reduce  the  Army’s  pro¬ 
curement  objective  by  79  percent  and  thus  are  the  root  causes  of  the  Nunn-McCurdy 
breaches  in  PAUC  and  APUC.  The  rationale  for  reducing  the  quantity  from  30,000  to 
6,264  units  includes  analysis  that  incorporated  a  change  in  operational  concept  that 
significantly  reduced  quantity  requirements.  Specifically,  there  was  a  change  in  the 
manner  in  which  the  artillery  is  used,  and  the  precision  of  the  Excalibur  unit  meant 
that  fewer  units  would  be  needed.  Additionally,  the  Munitions  Mix  Study  evaluated 
Excalibur  within  the  context  of  other  Army  systems  that  also  provide  precision  fires 
capability,  rather  than  as  a  stand-alone  (independent)  precision  fires  capability.  Afford¬ 
ability  also  appears  to  have  been  a  factor,  particularly  in  light  of  the  increased  pressure 
on  the  Army  procurement  budget. 

Table  2.2  shows  the  PARCA  root  cause  matrix  for  Excalibur.  The  cells  where  text 
is  present  indicate  that  the  factor  in  the  first  column  was  active  during  the  time  period 
indicated  in  the  column  heading  (first  row).  The  matrix  summarizes  the  narrative  given 
above. 

•  The  original  May  1997  cost  estimates  appear  to  have  been  highly  inaccurate. 

•  The  initial  SAR  cost  estimates  were  not  much  more  accurate,  as  evidenced  by 
unit  costs  being  unaffected  by  the  20  percent  quantity  reduction  in  2003  and  the 
subsequent  additional  51  percent  quantity  reduction  in  2004. 

•  A  concept  and  technological  change  appears  to  have  occurred  for  the  Excalibur 
system  between  the  original  solicitation  and  the  contract  award  to  Texas  Instru¬ 
ments  in  January  1998. 

•  The  high  level  of  performance  envisioned  for  Excalibur  turned  out  to  be  feasible. 


20 


See  the  May  12,2010,  ADM;  July  6,2010  Program  Deviation  Report;  and  Congressional  Notification  Letters. 
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Table  2.2 

PARCA  Root  Cause  Narrative  Matrix  for  Excalibur 


MS  I/ll 

May 

1997 

ACAT  1 
2002 

APB 

October 

2004 

APB 

Revisions 

September 

2007 

ADM 

May 

2010 

Baseline  issues 

Unrealistic  estimates 
for  cost  or  schedule 

May  1997  estimates 
highly  inaccurate 

Estimate  not 
reflective 
of  current 
system 

Immature  technology; 
excessive  manufactur¬ 
ing,  integration  risk 

Concept  and 
technology  change 
from  solicitation 
to  contract 
award  with  Texas 
Instruments  in 
January  1998 

Unrealistic  performance 
expectations 

Execution  issues 

Changes  in  procurement 
quantity 

Reduced 
10%  to 
61,483  from 
76,408 

Reduced 
51%  to 
30,000  from 
61,483 

Reduced 

79%  to 

6,264  from 
30,000 

Inadequate  funding/ 
funding  instability 

Addition 
of  Block  IB 
unfunded 

Unanticipated 
design,  engineering, 
manufacturing,  or 
technical  issues 

Minor 

technical 

issues 

Minor 

technical 

issues 

Minor 

technical 

issues 

Minor 

technical 

issues 

Poor  performance  of 
government  or  contract 
personnel 

X 

Other 

Urgent 

operational  need 
to  OEF/OIF  caused 
production 
acceleration  and 
more  Increment 
la  rounds 

•  The  quantity  reductions  from  the  original  200,000  to  76,408  appear  to  have 
been  driven  by  affordability  concerns. 

•  A  subsequent  reduction  in  the  Army’s  procurement  objective  from  76,408  to 
61,483  was  the  result  of  a  capabilities-based  requirements  analysis  in  2004  that 
recommended  the  61,483  quantity.  The  Army  Munitions  Mix  Study  is  cited  by 
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the  Army  as  the  capabilities-based  analysis  supporting  the  most  recent  79  percent 
quantity  reduction  to  6,264  units. 

•  The  Excalibur  program  experienced  relatively  minor  technical  issues  from  2002 
to  the  present.  These  issues  include  replacement  of  an  IMU  vendor  and  survivabil¬ 
ity  of  the  electronics  when  a  round  was  fired.  These  technical  problems  contrib¬ 
uted  to  some  cost  growth  but  were  not  significant  factors  in  the  Nunn-McCurdy 
breach. 

•  The  performance  of  government  and  contractor  personnel  appears  to  not  have 
been  an  issue,  with  the  possible  exception  of  the  IMU  vendor  replacement. 

•  Finally,  the  only  other  factor  influencing  Excalibur  unit  cost  growth  is  the  vali¬ 
dated  urgent  operational  need  for  OEF/OIF,  which  caused  production  to  be  accel¬ 
erated  and  the  production  of  more  Increment  la  rounds  than  originally  planned. 

Caveats  and  Future  Risks 

Additional  documentary  evidence  would  increase  confidence  in  the  root  cause  analysis 
described  above.  Specifically,  the  rationale  for  the  79  percent  reduction  in  the  Army’s 
procurement  objective  from  30,000  units  to  6,264  units  is  required.  The  Concepts 
Analysis  Agency  of  the  U.S.  Army  study  refers  to  a  “minimum  buy”  of  Excalibur  with¬ 
out  defining  this  term  and  without  an  explanation  of  how  it  was  calculated.  Insights 
into  how  this  figure  was  determined  would  bolster  the  analytic  support  of  the  root 
cause  analysis. 

In  addition,  the  CAA  charts  show  that  a  unit  cost  of  $40,000  was  used  for  the 
analysis.  The  CAA  study  does  not  indicate  what  year  dollars  the  $40,000  figure  is  in, 
but  it  is  not  consistent  with  any  PAUC  or  APUC  uncovered  to  date.  An  explanation  of 
why  this  unit  cost  was  used  and  how  it  was  calculated  or  a  reference  for  it  would  help 
explain  how  analysis  based  on  a  seemingly  different  unit  cost  supports  the  quantity 
reduction. 

Marine  Corps  buys  of  Excalibur  appear  to  have  been  in  the  1,000-unit  range. 
When  the  Army  procurement  quantities  were  in  the  30,000  range  or  higher,  the 
Marine  Corps  buy  would  have  represented  less  than  4  percent  of  the  total  and  could 
have  been  expected  to  have  had  very  little  effect  on  the  Excalibur  unit  cost.  However, 
given  that  the  Army  buy  is  in  the  7,000-unit  range,  the  Marine  Corps  buy  of  about 
1,000  units  represents  a  much  larger  portion  of  the  total  production  and  hence  should 
have  a  noticeable  effect  on  unit  cost.  When  the  Marine  Corps  buy  is  considered  along 
with  the  foreign  military  sales  quantities,  then  the  sum  of  these  external  quantities  can 
logically  be  expected  to  have  a  significant  effect  on  unit  cost.  Additional  documentary 
evidence  is  needed  to  explain  how  the  Marine  Corps  and  foreign  military  sales  are 
being  accounted  for  to  gain  a  full  understanding  of  the  true  unit  cost  of  Excalibur. 

Finally,  Army  budget  documents  indicate  that  the  Army  has  received  supplemen¬ 
tal  funding  to  support  the  validated  urgent  operational  need.  Additional  documenta¬ 
tion  that  provides  a  complete  accounting  for  all  funds,  regardless  of  origin,  expended 
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or  planned  to  be  expended  for  Excalibur  rounds  would  help  provide  a  full  understand¬ 
ing  of  the  true  unit  cost  of  Excalibur  rounds.  Again,  those  data  were  not  available  to 
RAND  in  the  allotted  time  for  this  RCA. 

However,  as  noted  above,  our  experience  in  analyzing  events  with  regard  to 
Excalibur  RCA  findings  led  RAND  to  develop  new  techniques  to  assess  the  relative 
contribution  of  risk  to  a  program  from  its  various  and  often  complex  component  sys¬ 
tems.  Chapters  Four  and  Five  of  this  report  present  complementary  views  for  examin¬ 
ing  risk  of  program  failure  within  these  complex  systems.  Chapter  Four  describes  an 
exercise  for  using  the  technical  complexity  of  a  program  and  its  level  of  detail  to  assess 
risk  and  identify  the  most  critical  features  of  a  program.  It  shows  that  by  narrowing 
the  scope  of  the  decisionmaker’s  review  to  a  handful  of  the  most  critical  components, 
the  incremental  technical  risks  that  may  lead  to  major  failures  could  be  identified  and 
possibly  averted. 

Chapter  Five  demonstrates  a  process  of  taking  the  more  narrowly  defined  review 
path,  crafted  as  a  result  of  the  exercise  outlined  in  Chapter  Four,  to  explore  the  tech¬ 
nical  level  of  risk  within  the  most  critical  components.  Using  the  Excalibur  as  an 
example,  Chapter  Five  demonstrates  that  by  focusing  the  decisionmaker  on  the  IMU 
and  the  GPS  receiver  components,  the  incremental  problems  that  ultimately  became 
detrimental  to  the  Excalibur  system  would  have  been  visible  early  in  the  program’s 
history.  By  identifying  the  incremental  problems  that  plague  the  most  critical  com¬ 
ponents,  decisionmakers  might  have  been  able  to  correct  the  program’s  path  before  it 
breached.  Considered  together,  Chapters  Four  and  Five  present  a  process  for  identify¬ 
ing  the  critical  components  and  then  determining  how  best  to  sift  through  their  data 
for  any  underlying  challenges  or  otherwise  unknown  weaknesses. 

Remaining  Risks  to  Monitor 

Two  risk  areas  in  our  analysis  warrant  monitoring:  Increment  lb  rounds  and  the  ability 
to  achieve  cost  goals  for  Increment  lb  using  planned  manufacturing  processes. 

The  source  for  Increment  lb  was  selected  at  the  end  of  August  2010,  with  a  con¬ 
tract  award  to  Raytheon  immediately  after.  Only  a  limited  number  of  rounds  were 
fired  as  part  of  the  competition.  Some  additional  testing  is  needed  to  get  a  better 
understanding  of  the  Increment  lb  round’s  actual  performance.  Although  the  develop¬ 
ment  risk  appears  to  be  low,  it  still  should  be  monitored. 

The  ability  to  achieve  Increment  lb  cost  goals  using  the  planned  manufactur¬ 
ing  processes  also  requires  attention.  The  most  significant  change  in  the  Increment  lb 
round  is  how  it  is  manufactured.  The  changes  are  intended  to  reduce  the  cost  of  Incre¬ 
ment  lb  production  to  half  that  of  Increment  la  rounds.  Changes  in  manufacturing 
processes  carry  some  risk  and  should  be  monitored.  If  the  expected  dramatic  reduction 
in  production  cost  cannot  be  attained,  then  the  current  unit  cost  estimates  will  be  too 
low  and  the  program  will  again  experience  growth  in  the  unit  cost  metrics. 
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The  Navy  Enterprise  Resource  Planning  Program 


This  chapter  considers  cost  growth  in  the  Navy  ERP  program.  It  begins  with  a  discus¬ 
sion  of  ERP  programs  in  general,  provides  an  overview  of  the  Navy  ERP,  and  presents 
its  cost  history,  including  increases,  program  changes,  and  schedule  delays.  It  then 
assesses  the  program’s  risk  and  draws  some  policy  lessons.  Because  the  ERP  was  not 
a  Nunn-McCurdy  breach,  more  time  was  available  to  study  it  than  was  available  for 
the  other  RCAs.  Thus,  the  analysis  considers  other  aspects  of  the  program  or  goes  into 
greater  depth  than  was  possible  for  programs  with  short  congressional  deadlines. 


Enterprise  Resource  Planning  Programs 

An  ERP  program  is  a  software  suite  designed  to  provide  an  organization  with  data  that 
can  be  aggregated  across  its  “enterprise.”  By  enforcing  standard  data  definitions  across 
an  enterprise,  the  software  collects  data  in  a  format  inherently  amenable  to  aggrega¬ 
tion.  The  redefined  and  formatted  data  allow  organizations  to  measure  their  processes 
and  results  in  ways  that  their  discrete  management  and  related  information  technology 
systems  could  not,  thus  facilitating  a  synoptic  view.  ERP  implementation,  as  a  result, 
moves  the  burden  of  work  from  (blue  collar)  data  entry  to  (white  collar)  data  analysis. 

Today,  ERPs  are  sold  generally  as  prepackaged,  commercially  available  software 
intended  to  merge  data  seamlessly  from  a  multitude  of  sources.  In  its  current  form, 
ERP  software  has  been  built  to  accommodate  a  set  of  standard  organizational  func¬ 
tions;  acquirers  of  the  software  can  select  the  prepackaged  capability  that  most  resem¬ 
bles  their  business.  Once  acquired,  this  software  is  then  configured  further  to  match 
the  specific  business  processes  of  the  user. 

It  is  easy  to  think  of  an  ERP  as  just  another  information  technology  acquisition, 
but  that  is  a  misleading  perspective.  Because  of  the  “enterprise”  nature  of  the  ERP 
program,  implementing  one  effectively  invariably  requires  two  steps:  understanding 
existing  business  processes  well  enough  to  define  accurately  the  nonstandard  data  that 
exist  currently,  and  re-engineering  business  process  as  to  best  practices  and  thereby 
producing  standard  data  in  the  future.  The  first  step  is  crucial  for  understanding  thor¬ 
oughly  the  reason  data  exist  in  the  current  format  and  for  eliciting  what  processes  are 
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essential  for  maintaining  an  efficient  and  effective  business  structure.  (Divergent  data 
often  indicate  that  extant  business  processes  do  not  correlate  well.)  End  results  of  the 
second  step  differ  for  each  organization  that  implements  an  ERP;  an  inherent  trade-off 
exists  between  software  customization  and  business  process  alteration,  with  the  latter 
offering  a  greater  degree  of  standardization. 

End-to-end  business  processes  must  be  standardized  to  create  data  that  can  be 
aggregated  from  disparate  organizational  components.  By  acting  as  a  tool  to  hold  busi¬ 
ness  data,  the  ERP  also  serves  as  a  forcing  function  to  identify  nonstandard  processes 
within  its  domain.  These  nonstandard  processes  can  be  altered  to  conform,  or  the  ERP 
software  can  be  modified  to  accommodate  a  one-off  solution.  Thus,  to  implement  an 
ERP,  organizations  must  necessarily  understand  not  only  what  their  business  processes 
are  but  also  what  they  ought  to  become.  Self-understanding  is  never  easy,  and  this  pro¬ 
cess  is  particularly  challenging  in  an  institution  such  as  the  U.S.  Navy,  which  is  very 
large,  very  old,  and  uses  complex  processes  representing,  in  part,  many  deep  cultural 
norms,  which  vary  from  one  command  to  another. 

Therein  lies  an  important  circularity.  Estimating  the  cost  of  acquiring  and,  more 
importantly,  implementing  an  ERP  requires  understanding  current  business  processes 
(as  well  as  how  they  might  evolve).  But  a  major  cost  of  implementing  an  ERP  is  doing 
precisely  that:  understanding  one’s  business  processes  well  enough  to  standardize  them 
for  the  ERP  program.  In  other  words,  to  estimate  the  cost  of  program  implementation 
requires  actually  implementing  at  least  the  front  end  of  the  program.  ERP  implementa¬ 
tion  includes  acquisition  of  the  “tool”  and  business  process  alteration,  yet  the  ability  to 
estimate  the  cost  of  business  practice  change  depends  heavily  on  the  knowledge  of  the 
individuals  involved  in  the  program.  The  current  difficulties  faced  by  the  Department 
of  Defense  (DoD)  in  implementing  ERP  programs  on  time  and  under  budget  reflect, 
in  part,  this  dilemma. 

The  Navy  ERP  program,  initiated  in  2003  and  fully  started  in  2004,  was  designed 
to  serve  as  the  technical  backbone  for  the  maintenance,  financial,  and  supply  functions 
of  the  Navy.  Although  it  is  currently  nearing  complete  implementation,  the  history  of 
the  program  reveals  that  its  cost  has  risen  and  its  schedule  has  lengthened  beyond  origi¬ 
nal  estimates.  As  a  result  of  increased  scrutiny  on  the  government’s  acquisition  of  ERP 
systems,  this  chapter  attempts  to  shed  light  on  the  root  causes  for  Navy  ERP  problems 
based  on  standard  MDAP  metrics;  it  also  addresses  the  topic  of  whether  MDAP  met¬ 
rics  are  the  appropriate  indicators  of  ERP  program  problems. 


Overview  of  the  Navy  ERP 

The  Navy’s  ERP  program  had  its  genesis  in  the  late  1990s,  when  the  combination  of 
shrinking  budgets  and  the  added  turbulence  created  by  base  closures  and  other  realign- 
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ments  within  the  Navy  prompted  officials  to  turn  to  a  new  class  of  software,  ERP  soft¬ 
ware,  as  a  way  to  gain  mastery  over  its  support  functions. 

Although  the  purported  purpose  of  the  pilot  ERP  varies  based  on  stakeholder 
viewpoint,  four  purposes  appear  fundamental. 

•  The  first  was  to  gain  a  modernized  capability  to  manage  System  Commands’ 
(SYSCOMs’)  post-1995  Base  Realignment  and  Closure  (BRAC)  consolidations 
and  realignments.  Most  legacy  information  systems  were  written  in  COBOL, 
which  was  becoming  increasingly  expensive  to  maintain.  BRAC  necessitated  a 
major  overhaul  of  existing  management  and  information  systems,  thus  providing 
an  opportunity  to  upgrade  to  an  ERP  program. 

•  The  second  was  to  start  the  process  toward  gaining  a  global  visibility  over  the 
data  in  the  Navy’s  support  base,  notably  financial  records,  supply,  and  repair. 
This  visibility  would  allow  for  better  informed  policy  analysis  to  provide  more 
efficient  operational  decisions,  driven,  in  part,  by  DoD-wide  pressure  to  improve 
management. 

•  The  third  was  to  achieve  “clean  financials,”  a  completely  reconciled  picture  of 
accounts,  assets,  disbursements,  receivables,  and  so  on.  This  goal  was  pursued  to 
achieve  conformance  with  the  Chief  Financial  Officers  (CFO)  Act  of  19901  and 
because  cleaner  accounts  allowed  program  and  financial  personnel  to  be  shifted 
from  tedious  reconciliation  duties  to  putatively  more  productive  analytic  tasks. 

•  The  fourth  was  to  liquidate  negative  wedges.  These  were  created  in  the  later  half 
of  the  1990s  as  a  function  of  the  Defense  Management  Review  process  under 
Secretary  of  Defense  Cheney.  This  review  postulated  savings  opportunities  and 
reduced  budget  top-lines  in  an  anticipatory  manner  as  an  incentive  to  command 
performance. 

From  the  outset,  the  Navy  realized  that  the  challenges  of  ERP  program  imple¬ 
mentation  were  going  to  differ  substantially  from  those  experienced  by  the  private 
sector  because  of  its  size  and  diverse  scope  of  business  functions.  It  therefore  authorized 
four  of  its  SYSCOMs  to  start  pilot  ERP  programs.  Table  3.1  lists  each  SYSCOM,  its 
pilot,  what  each  was  designed  to  do,  and  its  estimated  cost  (bear  in  mind  that  roughly 
half  of  these  costs  were  for  day-to-day  operating  expenses).  The  Navy  also  recognized 
the  inherent  complexity  of  this  project.  The  number  of  users  was  large  ("140,000),  a  sig¬ 
nificant  amount  data  had  to  be  converted  and  legacy  data  cleansed,  processes  included 
over  1,300  unique  transactions,  the  amount  of  data  involved  exceeded  15  terabytes, 
and  the  transaction  volume  was  large — over  32  million  transactions  per  month. 


1  For  more  information,  see  Public  Law  101-576,  Chief  Financial  Officers  Act  of  1990,  November  15,  1990,  or 
General  Accountability  Office,  The  Chief  Financial  Officers  Act:  A  Mandate  for  Federal  Financial  Management 
Reform,  September  1991. 
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Table  3.1 

ERP  Original  Pilot  Efforts 


Command 

Name 

Purpose 

Cost3 

Naval  Air  Systems  Command 
(NAVAIR) 

SIGMA 

Program  management,  contracting, 
financial,  and  time/attendance 

$215.9 

Naval  Supply  Systems  Command 
(NAVSUP) 

SMART 

Supply  management  for  the  LM-2500 
engine  and  the  EA-6 

$346.6 

Space  and  Naval  Warfare  Systems 
Command  (SPAWAR) 

CABRILLO 

Financial  management 

$67.4 

Naval  Sea  Systems  Command 
(NAVSEA) 

NEMAIS 

Intermediate  maintenance  management 

$414.6 

SOURCE:  Government  Accountability  Office,  DoD  Business  Systems  Modernization:  Navy  ERP  Adherence 
to  Best  Business  Practices  Critical  to  Avoid  Past  Failures,  Washington,  D.C.:  U.S.  Government  Printing 
Office,  GAO-05-858,  September  2005. 

a  Millions  of  dollars;  through  FY  2004. 

In  2003,  the  Navy  concluded  that  the  pilot  programs,  which  viewed  their  “enter¬ 
prise”  boundaries  at  the  SYSCOM  level,  had  run  their  course  and  that  it  was  time  to 
merge  their  results  into  a  converged  Navy  ERP  program,  whose  process  standardiza¬ 
tion  would  span  SYSCOMs.  Part  of  the  impetus  was  that  the  Navy  had  grown  more 
confident  in  its  understanding  of  ERP  programs.  Another  factor  may  have  been  dic¬ 
tates  from  the  OSD  Comptroller’s  office  to  the  military  departments  requiring  that  the 
latter  re-engineer  their  systems  to  produce  clean  financials.  2  The  Navy  concluded  that 
the  governance  structure  of  the  separate  ERP  programs  in  the  various  SYSCOMs — 
operating  different  schema  with  processes  optimized  only  at  the  SYSCOM  level — 
lacked  the  unity  and  coordination  required  for  these  mandates.  The  Navy’s  analysis 
indicated  that  integrating  the  pilot  ERP  systems  would  be  more  costly  and  less  effec¬ 
tive  than  starting  anew  with  a  single  Navy  ERP  program  office.  It  designed  a  hierarchi¬ 
cal  governance  structure  to  optimize  processes  across  SYSCOMs,  under  service-wide 
funding,  using  a  standard  software  platform  (produced  by  the  German  company  SAP, 
which,  it  turns  out,  was  the  software  backbone  selected  by  each  pilot  program  contract 
integrator  and  used  for  each  of  the  four  SYSCOMs’  pilots).3  To  develop  the  parameters 
of  the  Navy  ERP,  the  program  office  brought  ERP  teams  from  the  four  SYSCOMs  to 
work  together  and  determine  collectively  which  features  in  each  of  their  programs  were 


2  Although  the  pilot  ERPs  were  meant  to  enable  cleaner  financials,  it  was  known  that  they  would  not  produce 
auditable  records  by  themselves. 

3  Analytically,  it  could  have  been  useful  to  analyze  the  Navy  ERP  as  a  conglomeration  of  each  SYSCOMs  ERP, 
each  with  its  own  budget  and  milestone.  Such  a  record  may  have  indicated  whether  one  or  another  SYSCOM 
had  a  greater  cost  growth  or  a  longer  schedule  slip  than  its  counterparts — something  that  might  have  shed  light 
on  whether  differences  in  implementation  among  SYSCOMs  may  have  been  correlated  with  better  or  worse  per¬ 
formance.  Elowever,  the  Navy  does  not  keep  the  ERP  budget  that  way,  in  large  part  because  there  were  many 
functions,  such  as  help  desks,  that  are  common  across  programs. 
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worthwhile  and  which  functions  would  be  done  in  which  way.  The  result,  the  Navy 
argues,  is  a  “best-of-breed”  design  within  the  SYSCOM  environment. 

It  is  important  to  note  that  the  plan  from  2004  was  to  shut  down  legacy  systems 
as  the  Navy  ERP  went  live.  This  action  forced  future  Navy  ERP  use  and  thus  may  have 
increased  user  participation  in  the  acquisition  process.  It  reduced  cost  by  eliminating 
functionally  redundant  systems. 

Almost  immediately  after  the  Navy  ERP  baseline  was  established  (August  2004), 
the  program  was  buffeted  by  exogenous  turbulence  in  the  form  of  the  2005  round  of 
BRAC.  Less  important  were  the  details  of  that  recommendation  (which  went  to  Con¬ 
gress  for  ratification  in  late  2005)  than  the  Navy’s  decision  to  restructure  its  entire 
maintenance  activity  as  a  result. 


Cost  History 

Technically,  the  Navy  ERP  program  has  breached  Nunn-McCurdy  limits  (assum¬ 
ing  such  limits  were  meant  to  be  applied  to  a  Major  Automatic  Information  System 
[MAIS]  program).  The  program  acquisition  costs  were  significantly  more  than  esti¬ 
mated  in  August  2004  (the  date  of  its  first  baseline),  the  roll-out  of  the  first  increment 
went  live  a  year  or  two  later  than  planned,4  and  the  capabilities  of  the  current  program 
of  record  are  less  inclusive  than  those  originally  promised  in  the  baseline.  Yet  closer 
inspection  reveals  that  the  program  has  not  experienced  perpetual  turbulence.  Instead, 
estimates  of  cost  and  schedule  have  stabilized  in  a  predictable  manner  since  a  program 
re-baseline  in  December  2006. 

Table  3.2  illustrates  the  estimated  program  cost  (in  constant  2004  dollars)  at  four 
time  points:  August  2004  (the  baseline),  December  2006  (the  re-baseline),  late  2008, 
and  late  2010.  Costs  rose  23  percent  between  the  baseline  and  the  re-baseline;  they 
then  fell  7  percent  through  the  end  of  2008  (as  a  result  of  reduced  scoping  decisions) 
and  rose  3  percent  as  of  September  30,  2010. 

Of  note,  almost  all  of  the  estimate  fluctuation  occurs  within  the  acquisition 
(RDT&E,  procurement,  acquisition  operations  and  maintenance  [O&M],  and  work¬ 
ing  capital  fund-capital  budget  and  operating  budget)  line  item,  even  though  these 
operations  and  support  (O&S)  costs5  represent  over  half  of  the  total.  Table  3.3  com¬ 
pares  the  changes  in  the  two  from  one  period  to  another.  The  RDT&E  account,  at 
least  initially,  had  even  larger  fluctuations,  almost  doubling  between  the  baseline  and 
the  re-baseline. 


4  The  go-live  date  for  every  SYSCOM  was  staggered  so  that  each  one  went  live  in  a  different  fiscal  year.  Further¬ 
more,  within  SYSCOMs,  certain  functions  went  live  at  different  times.  Hence  the  one-to-two  estimate  applies 
over  the  range  of  functions. 

5  Life  cycle  cost  estimates  include  ten  years  past  the  date  of  full  operational  capability.  This  definition  has 
remained  consistent  throughout  the  life  of  the  Navy  ERP  thus  far. 
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Table  3.2 

Navy  ERP  Costs  (millions  of  constant  2004  dollars) 


August 

2004 

December 

2006 

December 

2008 

September 

2010 

RDT&E 

14S.1 

287.0 

297.1 

295.7 

Procurement 

75.4 

61.0 

61.5 

62.1 

Acquisition  O&M 

317.9 

462.0 

310.9 

332.4 

Working  capital  fund — capital 

88.6 

12.0 

12.5 

17.0 

Working  capital  fund — operating 
budget 

168.0 

160.2 

217.9 

O&S 

1,005.9 

1,024.0 

1,028.4 

1,002.9 

Total 

1,632.9 

2,014.0 

1,870.6 

1,928.0 

%  change 

23.3 

-7.1 

3 

Table  3.3 

Navy  ERP  Costs  and  Percentage  Change  (millions  of  constant  2004  dollars) 


August 

2004 

December 

2006 

December 

2008 

September 

2010 

Acquisition 

627.0 

990.0 

842.2 

925.1 

%  change 

+  58 

-  15 

+  10 

O&S 

1,005.9 

1,024.0 

1,028.4 

1,002.9 

%  change 

+  2 

+  0.5 

-2.5 

Some  of  these  differences  in  cost  increases  over  time  may  well  be  a  function  of 
cost  allocation  decisions  by  the  Navy  as  program  execution  progressed.  Some  of  the 
cost  allocation  methodology,  in  turn,  is  subsumed  in  the  release  changes  reflected  in 
Table  3.4  and  the  attending  sources  of  financing  for  O&S.  There  is  no  clear  record  of 
these  adjustments. 

Thus,  this  root  cause  analysis  focuses  on  internal  program  misconceptions  and 
external  shocks  that  resulted  in  initial  estimation  error.  These  dual  phenomena — the 
cost  estimate  increase  and  schedule  slippage  that  occurred  between  2004  and  2006 — 
coupled  with  the  relative  stability  of  the  program  since  then  and  the  fact  that  the  vari¬ 
ous  SYSCOMs  have,  in  fact,  gone  live  are  what  are  to  be  explained. 
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Table  3.4 

Go-Live  Dates  for  Release  1.0  at  Navy's  SYSCOMs 


As  of 

August  2004 

As  of 

December  2006 

As  of 

September  2010 

NAVAIR 

FY  08 

FY  08 

NAVSUP 

FY  09  (2nd  quarter) 

FY  09 

SPAWAR 

FY  08  (3rd  quarter) 

FY  10 

NAVSEA 

FY  10  (2nd  quarter) 

FY  11 

Program  Changes 

The  Navy’s  decision  to  shift  from  intermediate  maintenance  activities  to  regional 
maintenance  activities  was  profound,  and  as  a  result,  unsettled  the  design  and  char¬ 
acteristics  of  the  entire  business  process  of  the  maintenance  activity.  In  2004,  the 
Navy  ERP  program  had  planned  an  end-to-end  system  that  would  integrate  main¬ 
tenance  into  finance  and  supply.  The  plan  (blueprint  in  ERP  parlance)  predicated 
intermediate-level  maintenance  as  the  first  release.  Yet  the  maintenance  activities 
themselves  were  no  longer  stable.  Although  the  Navy  ERP  program  office  maintained 
risk  mitigation  contingency  plans  that  included  schedule  delays  resulting  from  exter¬ 
nal  program  events,6  the  plans  did  not  include  contingencies  for  institutional  organi¬ 
zational  changes.  The  Navy  ERP  program  office  recognized  the  need  for  substantial 
blueprint  revisions  to  accommodate  the  new  organization  of  the  Navy’s  maintenance 
activities.  Unfortunately  for  planners,  the  Navy’s  maintenance  architecture  did  not 
converge  quickly  enough  to  permit  a  solid,  modified  blueprint.  The  latter  effort  there¬ 
fore  took  a  considerable  amount  of  time  (a  year  or  more)  during  which  the  meter  was 
running,  so  to  speak.  This  activity  was  reflected  in  the  doubling  of  RDT&E  expendi¬ 
tures  between  the  original  baseline  and  the  new  baseline  generated  in  December  2006. 

Table  3.5  reflects  the  effect  of  these  changes,  over  time,  in  the  standard  PARCA 
root  cause  narrative  methodology. 

Cost  Increases 

Although  the  blueprint  revision  was  expensive,  it  does  not  appear  to  account  for  the 
entire  $400  million  difference  in  cost  estimates  between  the  first  and  second  baseline. 
The  scope  of  the  Navy  ERP  program  after  the  second  baseline  was  no  greater  than 
it  was  after  the  first  baseline;  in  fact,  it  was  putatively  smaller.  Thus,  the  most  logi¬ 
cal  inference  is  that  the  Navy  ERP  program  office  took  the  opportunity  presented  to 
them  by  BRAC  to  offer  a  new  baseline  that  incorporated  lessons  learned  regarding  the 


6  Risk  mitigation  planning  concepts  are  discussed  briefly  in  the  2004  Navy  ERP  ORD.  RAND  has  not  seen 
more  detailed  plans.  Note  that  the  Navy  ERP  ORD  source  of  this  information  is  likely  not  available  to  the  public. 
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Table  3.5 

PARCA  Root  Cause  Narrative  for  the  Navy  ERP 


Year  from  MS  B  and  Fiscal  Year 


B  +1  +2  +3  +4  +5  +6 

2004  2005  2006  2007  2008  2009  2010 


Baseline  issues 

Unrealistic  estimates  for  X  X 

cost  or  schedule 

Immature  technology; 
excessive  manufacturing, 
integration  risk 

Unrealistic  performance  X 

expectations 

Execution  issues 

Changes  in  procurement  X  X 

quantity 

Inadequate  funding/ 
funding  instability 

Unanticipated 
design,  engineering, 
manufacturing,  or 
technical  issues 

Poor  performance  of 
government  or  contract 
personnel 

Unanticipated  exogenous  X  X 

business  practice  changes 


cost  of  the  Navy  ERP.  In  other  words,  had  the  BRAC  never  occurred,  the  Navy  ERP 
would  have  likely  overrun  its  initial  cost  estimate,  although,  almost  certainly,  by  less 
than  $400  million. 

The  second  baseline  was  generated  in  December  2006  prefatory  to  the  Milestone 
C  decision  of  September  2007.  In  the  re-baselined  version,  the  Navy  ERP  program 
office  moved  finance  to  the  head  of  the  capability  release  queue  (release  1.0),  followed 
by  supply  (release  1.1)  and  maintenance  (release  1.2).  This  reorientation  reflected  the 
low  confidence  the  program  had  that  the  architecture  of  the  Navy’s  maintenance  activ¬ 
ities  would  stabilize  quickly.  This  reorientation  also  included  back-shop  revisions  as  the 
scope  of  the  finance  release  was  broadened  to  include  some  of  the  work  necessary  for 
whichever  capability  came  first,  initially  planned  for  the  maintenance  release. 

Additionally,  the  Navy’s  ERP  program  office  altered  the  original  release  concept 
to  include  a  spiral  approach  by  planning  to  release  capabilities  in  phases.  It  further 
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divided  the  releases  to  the  user  groups.  In  this  way,  even  as  capability  was  portioned 
according  to  releases  (maintenance,  finance,  then  supply),  each  release  had  a  different 
go-live  date  for  each  SYSCOM,  often  a  full  year  apart. 

As  it  became  clear  that  the  maintenance  activity  would  not  stabilize  sufficiently 
to  allow  inclusion  in  the  Navy  ERP  whose  full  development  decision  was  planned  for 
March  2011,  the  ERP  program  office  dropped  maintenance  entirely  from  the  program 
of  record  in  late  2008,  with  corresponding  decreases  in  the  cost  of  the  Navy  ERP. 
However,  program  managers  were  careful  to  build  in  hooks  from  the  finance  and 
supply  activities  to  some  future  maintenance  activity.  This  way,  if  and  when  the  latter 
did  stabilize,  the  Navy  ERP,  by  then  well  established,  could  interface  with  some  future 
maintenance  component  of  the  ERP  without  significant  revisions  in  the  overall  blue¬ 
print  or  in  the  configuration  of  the  rest  of  the  Navy  ERP. 

Schedule  Delays 

The  first  and  major  schedule  slippage  was  likely  associated  with  the  2005  BRAC-related 
disruption  when  the  program  essentially  remained  static  for  about  a  year.  The  original 
IOC  estimate  for  the  ERP  program  (or  as  it  was  known  then,  the  ERP  Convergence 
program)  was  the  first  quarter  of  FY  2007  for  release  1.0  that  was  originally  going  to 
be  maintenance.  Such  plans  did  not  distinguish  between  SYSCOMs.  The  actual  IOC 
was  a  year  later,  for  just  one  SYSCOM  (NAYAIR)  and  involved  a  redefined  release  1.0 
of  finance.  One  way  to  understand  the  slippage  is  that  the  earliest  module  to  was  not 
implemented  but  that  subsequent  modules  reached  their  earliest  IOC  as  scheduled  but 
not  for  the  whole  Navy;  the  other  SYSCOMs  had  staggered  and  later  IOCs. 

Even  after  costs  stabilized  following  re-baseline,  there  was  continued,  albeit  minor 
slippage  in  schedules  after  that  point.  Several  explanations  for  this  can  be  offered.  As 
noted,  one  reason  is  that  the  program  office  concluded  that  staggered  go-live  dates 
for  each  of  the  four  SYSCOMs  would  be  easier  to  manage  than  having  two  or  more 
SYSCOMs  go  live  at  roughly  the  same  time.  Two,  closely  related,  is  the  realization 
that  ERP  programs  that  have  a  profound  effect  on  financial  accounting  should  go  live 
as  the  fiscal  year  begins  rather  than  mid-year.  Three,  more  subtle  but  indicative  of  the 
difference  between  an  information  technology  acquisition  and  an  ERP  acquisition,  the 
program  office  was  concerned  that  one  or  more  SYSCOMs  were  not  ready  to  receive 
their  ERP.  The  mismatch,  it  was  feared,  would  wreck  the  processes  that  the  ERP  was 
trying  to  improve;  customers  did  not  have  the  choice  of  putting  the  ERP  program  aside 
until  they  were  ready;  the  delivery  of  the  program  meant  that  legacy  systems  would  be 
turned  off  and  new  business  processes  would  begin.  By  contrast,  the  schedule  for  deliv¬ 
ery  of  an  air  platform  MDAP  would  not  be  so  sensitive  to  user  readiness;  were  one  to 
deliver  a  jet  before  users  were  ready,  the  worst  that  could  happen  was  that  it  would  not 
be  used  very  intensively  in  its  first  few  months  at  the  base. 
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RAND's  Assessment  of  Future  Navy  ERP  Program  Risk 

At  this  juncture,  the  Navy’s  ERP  can  be  said  to  be  a  qualified  technical  success.7  It  has 
been  implemented  at  three  SYSCOMs.  The  two  we  visited  can  document  the  advan¬ 
tages  they  have  gained  so  far.  NAVAIR,  for  instance,  has  compiled  a  set  of  indicators 
for  this  purpose:  e.g.,  timesheet  compliance,  contract  obligation  efficiency,  late  vendor 
payments,  pay  acceptance,  unrecorded  expenditures,  unpaid  balances,  travel  orders 
completed,  and  others.  Most  of  them  are  going  in  the  right  direction,  and  some  of 
them  have  exceeded  objectives.  NAVSUP,  so  far,  has  found  a  half-billion  dollar’s  worth 
of  cost  savings,  divided  equally  between  reduced  inventories  (because  NAVSUP  has 
better  global  visibility  into  where  its  inventory  sits  and  can  detect  items  in  oversup¬ 
ply  worldwide)  and  lower  information  system  maintenance  costs.  Indeed,  the  Navy 
was  able  to  turn  off  over  a  hundred  legacy  systems  when  its  ERP  came  online.  And 
all  this  does  not  include  the  harder-to-quantify  benefits  of  fewer  administrative  errors, 
the  ability  of  personnel  to  shift  from  one  job  to  another  without  having  to  learn  a  new 
system,  the  transition  of  the  SYSCOMs  from  transaction  commands  to  analytic  com¬ 
mands,  or  better  management  decisionmaking. 

Nevertheless,  there  is  still  a  fair  amount  of  optimization  and  fine-tuning  that 
has  yet  to  be  completed.  Unmatched  disbursements,  for  instance,  still  exist  within  the 
current  ERP,  and  the  cross-functional  value  to  the  Navy  (rather  than  to  individual 
SYSCOMs)  has  yet  to  be  documented.  Tensions  persist  between  the  various 
SYSCOMs,  which  want  to  optimize  the  ERP  to  their  unique  issues,  and  the  overall 
Navy,  which  wants  conformance  to  its  enterprise  model.  Although  today’s  financials 
and  audits  are  cleaner  than  they  used  to  be,  they  are  not  clean  in  the  absolute  sense 
and  may  never  be,  given  both  the  tendency  of  financial  accounting  to  lag  changes  in 
financial  policy  and  the  interrelationships  with  Department  of  Treasury  systems  and 
processes. 

What  Went  Right? 

Although  the  Navy  ERP  program  did  not  meet  initial  cost  and  schedule  estimates,  a 
re-baseline  after  two  years  proved  accurate  enough  to  predict  internal  milestones  and 
to  allow  users  to  report  satisfaction  with  the  delivery  of  the  program. 

Several  factors — many  of  which  were  common  to  experiences  in  the  private  sector 
successes  but  others  unique  to  the  Navy  process — may  have  contributed  to  relative 
program  success. 

First,  the  pilot  projects  served  as  a  de  facto  spiral  acquisition  approach  for  Navy 
ERP  implementation.  Granted,  they  were  not  cheap  and,  with  the  partial  exception 
of  NAVAIR’s,  not  an  insignificant  part  of  the  enduring  Navy  ERP  program  infra- 


7  RAND’s  analytic  conclusion  contrasts  with  GAO’s  conclusions  of  the  Navy  ERP  program  presented  in  reports 
published  by  GAO  in  2005,  2008,  2009,  and  2010.  RAND  used  these  reports  to  orient  its  research  and  ensure 
that  work  was  not  repeated  unnecessarily. 
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structure.  Yet  they  did  give  the  SYSCOMs  experience  using  an  ERP  and  helped  them 
understand  many  of  their  current  business  processes.  They  solved  some  data  confor¬ 
mance  and  business  process  issues.  They  also  provided  a  sense  of  what  changes  would 
have  to  be  made  to  accommodate  the  transitions  from  local  legacy  management  and 
information  systems  to  integrated  ERP  systems. 

Second,  cost-plus  contracting  allowed  the  government  to  subsume  part  of  the  risk 
associated  with  altering  its  own  business  practices  because  of  the  cost-plus  nature  of 
the  contract.8  The  program  office  is  convinced  that  a  contract  vehicle  that  shared  the 
cost  risk  between  the  government  and  the  contractor  was  necessary  for  getting  the  lat¬ 
ter’s  full  cooperation  on  a  project  for  which  the  government  had  little  a  priori  expertise. 
This  fostered  an  honest  give-and-take  between  the  contractor  and  the  Navy  essential 
to  understanding  the  Navy’s  business  processes  and  how  they  might  have  to  change  to 
accommodate  an  ERP. 

Third,  the  determination  to  minimize  the  customization  of  the  SAP  solution 
decreased  certain  types  of  technological  risks  to  cost  and  schedule.  NAVSUP  learned 
from  the  Defense  Logistics  Agency  experience  with  its  own  business  systems  imple¬ 
mentation  that  less  customization  had  several  major  advantages.  The  resulting  software 
would  be  less  expensive,  likely  have  fewer  bugs,  place  the  onus  of  software  upgrades  on 
the  software  provider,  and  better  align  business  processes  with  commercial  best  prac¬ 
tices.  Since  the  commercial  software  had  to  be  pre-approved  and  tested,  reliance  on 
it  could  help  ensure  a  speedier  program  implementation.  Failure  to  sharply  scrutinize 
requests  for  customization  might  allow  users  to  believe  that  they  could  maintain  their 
old  (and  often  incompatible)  business  practices,  and  refusal  to  allow  many  changes  was 
a  forcing  function  for  standardization.  NAVSUP  authorities  boast  that  only  2  percent 
of  all  Navy  transactions  required  departing  from  the  standard  SAP  transaction  set 
(NAVSUP’s  own  departure  rate  was  well  below  2  percent). 

Fourth,  the  interactive  governance  of  ERP  allowed  those  who  implemented  it  to 
develop  a  sense  of  staging  as  the  program  progressed.  Not  only  were  they  careful  not  to 
break  business  processes  in  the  course  of  implementation,  but  they  also  distinguished 
between  financial  systems  (where  a  “big-bang”  switchover  was  appropriate)  and  mate¬ 
rial  management  (where  such  a  switchover  was  not  necessary).  NAVSUP  staged  the 
latter  in  several  tranches,  with  each  tranche  mixing  in  hard  and  easy  problems  rather 
than  pushing  the  harder  problems  into  the  last  tranche.  They  also  established  20  sta¬ 
bilization  criteria  (a  quarter  of  which  were  deemed  major)  by  which  to  evaluate  the 
transition. 

Fifth  is  that  the  Navy  and  its  SYSCOMs’  leadership  showed  a  consistent  high-level 
interest  in  the  program.  This  support  proved  critical  for  maintaining  the  (as  described) 
painful  business  re-engineering  that  seemingly  suboptimized  command  processes.  The 
NAVSUP  commander  held  weekly  meetings  on  the  organization’s  implementation 


The  contract  was  a  combination  of  cost-plus-fixed-fee  and  cost-plus-award-fee. 
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process  and  managed  to  appear  in  person  at  most  of  them  (he  participated  in  the  rest 
via  telephone  or  video  teleconference).  Vice  Admiral  Lockhard,  the  NAVAIR  com¬ 
mander  (when  the  first  pilot,  entitled  SIGMA,  was  initiated),  was  also  a  strong  propo¬ 
nent  and  contributed  key  personnel  to  the  ranks  of  Navy  ERP  management. 


Concluding  Policy  Lessons 

The  rationale  for  applying  a  Nunn-McCurdy  test  to  an  ERP  acquisition  program  is, 
at  first  glance,  compelling.  These  are  expensive  programs,  a  larger  percentage  of  which 
fail,  often  spectacularly  compared  to  MDAPs.  Large  software  programs  almost  always 
cost  more  than  planned,  finish  later  than  scheduled,  and  deliver  less  than  promised. 
Several  colossal  government  system  acquisition  failures  have  become  virtual  legends: 
the  Federal  Bureau  of  Investigation’s  Virtual  Case  File  system,  the  Federal  Aviation 
Administration  (FAA)  Air  Traffic  Control  system  upgrade,  New  Jersey’s  Department 
of  Motor  Vehicles  system,  and,  within  DoD,  the  Defense  Integrated  Military  Human 
Resources  System.9  Typically,  when  such  systems  do  fail,  very  little  is  recoverable,  other 
than  hard-won  wisdom.  They  also  demonstrate  a  high  rate  of  failure  in  the  commercial 
world.10  ERP  failures  were  not  uncommon  in  the  private  sector  efforts  also. 

At  second  glance,  the  case  is  less  obvious.  The  basic  theory  behind  Nunn- 
McCurdy  is  that  the  inability  of  a  program  to  control  its  costs  and  keep  on  a  schedule 
is  a  sure  indicator  of  trouble.  Such  programs  often  appear  fine  in  reporting  metrics 
until  the  point  comes  when  they  are  not.  Rapid  intervention  is  then  required  to  deter¬ 
mine  whether  such  breaches  should  be  mitigated  by  program  changes,  or,  instead, 
whether  such  a  breach  is  the  first  of  many  signs  that  a  program  simply  cannot  perform. 
Either  way,  breaches  generally  indicate  that  the  program  needs  major  scope  changes 
or  its  costs  will  continue  to  escalate  out  of  control.  This  is  an  important  consideration 
for  hardware  MDAPs  because  the  bulk  of  all  costs  occur  during  the  production  rather 
than  the  development  phase.  By  contrast,  ERP-like  MAIS  programs  inherently  com¬ 
plete  an  abridged  acquisition  process,  where  software  development  and  software  pro¬ 
duction  occur  concurrently,  yielding  a  deployable  product.  As  such,  early  cost  increases 
may  be  the  only  cost  increases  if  other  factors  such  as  executive  leadership  support, 
contractor  compliance,  and  strong  governance  exist. 


9  For  the  FBI’s  Virtual  Case  File  system  see  Harry  Goldstein,  “Who  Killed  the  Virtual  Case  File?”  IEEE  Spec¬ 
trum,  Vol.  42,  No.  9,  September  2005,  pp.  24-35.  For  the  FAA’s  air  traffic  control  system,  see  Government 
Accountability  Office,  Air  Traffic  Control:  Immature  Software  Acquisition  Processes  Increase  FAA  System  Acquisi¬ 
tion  Risks,  Washington  D.C.,  GAO/AIMD-97-47,  March  1997.  For  the  Defense  Integrated  Military  Human 
Resources  System,  see  Kevin  McCaney,  “Readers  Offer  Stuffing  for  IT  Turkey,”  Federal  Computer  Week,  Decem¬ 
ber  1,  2009.  For  the  New  Jersey  Department  of  Motor  Vehicles  System,  see  Robert  Glass,  Software  Runaways: 
Monumental  Software  Disasters,  Upper  Saddle  River,  N.J.:  Prentice  Hall,  1997. 

10  Chris  Kanaracus,  “Biggest  Enterprise  Resource  Planning  Disasters  of  2010,”  PCWorld,  December  17,  2010. 
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Another  problem  with  extending  the  Nunn-McCurdy  criterion  is  the  unavoid¬ 
able  ambiguity  associated  with  counting  “eaches”  in  a  business  process  system — a  far 
smaller  concern  when  tracking  cost  growth  in  hardware  programs.  The  Navy  ERP,  for 
instance,  covers  a  large  number  of  people,  but  most  of  them  use  the  Navy  ERP  only 
for  time-and-attendance  worksheets.  The  bulk  of  the  Navy  ERP  costs,  however,  arise 
from  bringing  asset  and  financial  management  programs  within  the  logistics  com¬ 
munity  to  a  smaller  population  of  power  users.  Each  class  of  power  users,  in  turn, 
employs  different  levels  of  functionality  from  within  the  ERP  program.  This  is  more 
than  an  accounting  quibble.  The  Navy  ERP  has  been  deemed  sufficiently  successful 
that  moves  are  under  way  to  extend  it  to  shipboard  applications  and  other  functions 
outside  logistics  channels.  But  the  Navy  faces  the  difficult  choice  between  calling  an 
extension  a  new  program  start,  with  all  the  associated  paperwork  thereby  implied,  or 
simply  extending  the  Navy  ERP,  which  creates  ostensible  program  growth  according 
to  Nunn-McCurdy  criteria,  making  a  largely  successful  program  appear  to  be  a  failure. 

Consider,  therefore,  what  might  have  resulted  from  a  straightforward  application 
of  the  Nunn-McCurdy  test  to  the  Navy  ERP.  The  breach,  as  such,  would  have  arisen 
in  late  2006.  The  process  associated  with  the  breach  would  have  asked  whether  the 
Navy  ERP  was  under  control  or,  instead,  destined  to  explode.  Perhaps  the  program 
managers  could  have  successfully  argued  that  the  program’s  problems  were  created  by 
exogenous  factors  (although  some  early-course  estimate  correction  was  also  involved). 
In  retrospect,  however,  it  was  clear  that  the  Navy  ERP  was  not  headed  toward  disaster; 
the  program  has  been  quite  stable  since  then.  By  2007,  in  fact,  much  of  the  most  dif¬ 
ficult  work  and  “lessons  learned”  had  been  incorporated  into  the  new  baseline  as  the 
first  release  was  about  to  go  live  at  NAVAIR. 

More  broadly,  ERP  programs  are  not  stand-alone  procurements  in  the  same  sense 
that  a  jet  fighter  acquisition  or  even  a  network  defense  project  is.  The  major  share  of 
the  cost  is  associated  with  understanding  and  adapting  business  processes  to  the  busi¬ 
ness  model  built  into  the  software.  As  noted,  one  cannot  understand  the  cost  of  an 
ERP  without  understanding  the  enterprise  that  the  ERP  is  designed  to  re-engineer, 
but  understanding  that  enterprise  constitutes  a  major  share  of  the  overall  ERP  costs. 
As  for  alternative  cost-estimating  techniques,  notably,  parametric  cost  estimation  (e.g., 
in  terms  of  reports,  interfaces,  conversions,  and  extensions  objects),  accuracy  is  hostage 
to  the  great  variance  in  legacy  business  processes  from  one  organization  to  another. 
Finally,  the  number  of  people  in  the  Navy  who  knew  enough  to  establish  requirements 
was  quite  limited,  and  such  individuals  were  often  engaged  elsewhere.  The  Navy  ERP 
program  office  attempted  to  implement  a  clearly  defined  “implementation  cost-sharing 
scheme”  between  the  SYSCOMs  and  the  program  office  to  avert  some  of  the  moral 
hazard  endemic  when  offering  money  from  one  organization  to  another  for  business 
system  use.  It  helped  but  hardly  addressed  the  root  dilemma. 


CHAPTER  FOUR 
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A  primary  emphasis  in  the  performance  evaluation  of  an  MDAP  is  on  cost.  In  part 
this  is  because  unit  cost  growth  is  the  primary  trigger  for  a  Nunn-McCurdy  breach. 
This  is  clearly  demonstrated  by  the  organization  of  the  “speeding  ticket”  displayed  in 
Table  2.1.  However,  many  underlying  factors  contribute  to  program  performance  and 
risk  of  failure.  During  the  RAND  root  cause  analysis  of  the  six  programs  assigned  to 
date  for  review,  we  observed  a  number  of  approaches  used  to  characterize  the  nature  of 
program  risk  and  progress  that  were  applicable  DoD-wide.  These  various  approaches 
include  those  used  to  create  the  Program  Deviation  Report  that  describes  the  program’s 
cost  growth,  the  SAR,  as  well  as  approaches  taken  by  other  organizations  such  as  in  the 
GAO  Defense  Acquisitions  Assessments  of  Selected  Weapons  Programs  serial  reports. 

Although  these  approaches  may  be  useful  in  providing  a  rich  opportunity  for 
observation,  in  many  instances,  indications  that  problems  existed  were  not  sufficiently 
obvious  to  be  used  in  a  timely  manner  by  decisionmakers.  However,  the  most  impor¬ 
tant  details  are  not  often  presented  in  an  easy-to-use  form.  Decisionmakers  need  tools 
to  navigate  this  mass  of  bibliographic  data,  but  even  more  important  is  the  experi¬ 
ment  designed  to  help  a  decisionmaker  initially  chart  a  course  through  the  data.  Well- 
designed  exercises  could  help  make  more  obvious  the  critical  components  that  pose 
the  greatest  risk  of  program  failure.  With  this  information,  a  more  informed  decision¬ 
maker  might  make  more  timely  inquiries  into  the  performance  of  those  critical  fea¬ 
tures  before  problems  mount. 

When  unaddressed,  problems  originating  from  the  most  critical  features  of  a  pro¬ 
gram  threaten  the  life  cycle  of  the  program.  To  make  the  information  that  DoD  deci¬ 
sionmakers  need  more  timely  and  actionable,  decisionmakers  should  adopt  a  frame¬ 
work  for  thinking  about  the  program  features  ahead  of  attempting  a  deep  dive  into  the 
data  looking  for  problems.  An  initial  conceptual  framework  would  allow  a  decision¬ 
maker  to  quickly  determine  what  is  most  critical,  complex,  or  least  understood  of  the 
list  of  program  features.  Root  cause  analysis  as  an  element  of  Nunn-McCurdy  breaches 
focuses  on  program  acquisition,  but  the  process  used  by  analysts  to  structure  review 
criteria  could  also  be  used  by  decisionmakers  to  identify  total  ownership  cost  chal¬ 
lenges  and  critical  points  of  risk  in  the  complex  programs. 
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This  chapter  discusses  methods  to  identify  critical  features  to  help  decisionmakers 
focus  their  effort  to  uproot  various  program  risks  through  a  selective  screening  experi¬ 
ment.1  In  our  discussion  of  Excalibur  in  this  report,  we  review  critical  features  of  that 
program  through  monthly  technical  risk  reports  and  conclude  that  early  warning  signs 
were  available  that  could  have  signaled  the  program’s  trajectory  toward  failure.  Our 
analysis  and  the  findings  presented  in  our  Nunn-McCurdy  Breach  Program  Reports 
suggest  that  it  may  be  possible  to  develop  a  framework  that  would  enable  a  different 
review  of  programs  to  enhance  our  understanding  of  risk.  This  chapter  provides  an 
exercise  designed  to  help  decisionmakers  develop  the  type  of  initiating  hypotheses  that 
are  necessary  to  chart  a  path  through  the  bibliographic  data,  leading  the  decisionmaker 
to  the  critical  features  with  the  greatest  risk  to  the  program.  The  exercise  described  here 
is  an  approach  the  authors  felt  most  useful;  clearly,  others  may  be  designed. 


Measures  of  Merit 

The  exercise  described  in  this  chapter  incorporates  two  of  the  many  concepts  that 
root  cause  analysts  use  to  develop  an  initiating  hypothesis  about  why  a  program  has 
breached.  These  critical  concepts  include  the  program  system  complexity  and  the  level 
of  detail  available  about  various  technical  components.  Early  in  the  root  cause  analysis, 
the  review  team  assembles  a  time  line  of  the  program’s  relevant  history.  This  process  is 
discussed  in  detail  in  a  companion  forthcoming  report.2  At  this  early  stage,  the  review 
team  identifies  the  system  failures,  schedule  delays,  and  other  incremental  challenges 
that  led  to  the  program  failure  by  tracing  through  the  detailed  log  and  bibliographic 
data.  The  components  that  are  reviewed  to  develop  the  time  line  often  are  the  most 
complex  features  of  a  program.  In  the  past,  the  most  important  details  about  these 
complex  program  components  have  been  uncovered  in  the  least  visible  places — buried 
in  briefing  charts,  hidden  in  technical  log  files,  and  only  briefly  referred  to  in  monthly 
engineering  reports.  The  components  that  the  review  team  chooses  to  highlight  in  the 
program’s  time  line  are  often  the  most  complex  and  least  visible,  because  these  often 
contain  the  most  risk  to  the  program. 

We  propose  that  decisionmakers  use  a  “selective  screening  of  critical  components” 
process  to  identify  the  features  of  most  risk  to  the  program,  similar  to  the  process  of 
identifying  items  for  a  review  team  to  track  along  the  program’s  time  line.  To  identify 
the  components  with  the  most  potential  program  risk,  we  developed  a  methodology  for 
the  exercise  based  on  the  root  cause  analysis  process  to  relate  “measures  of  merit” — the 


1  The  examples  in  this  chapter  cover  many  of  the  other  programs  detailed  in  this  and  previous  publications  in 
the  RAND  RCA  series.  One  notable  exception,  the  Navy  ERP,  was  not  included  because,  as  a  software  system 
aimed  at  standardizing  Navy  business  operations,  the  program  is  not  an  MDAP. 

2  Irv  Blickstein  et  al.,  Methodologies  for  Analyzing  the  Root  Causes  of  Nunn-McCurdy  Breaches,  Santa  Monica, 
Calif.:  RAND  Corporation,  TR-1248-OSD,  forthcoming. 
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standard  measures  used  by  a  variety  of  Jane’s  publications3  to  describe  programs — to 
the  complexity  and  level  of  data  detail  available  for  specific  program  features.  What 
we  define  as  a  measure  of  merit  is  broadly  a  set  of  technical  components  that  con¬ 
tribute  to  a  measurable  process.  As  an  example  of  a  measure  of  merit,  the  Longbow 
Apache  AD-64A  (AB3)  power  plant  design  includes  two  General  Electric  T700-GE- 
701C  turboshafts,  which  are  each  rated  at  1,409  kW  for  10  minutes  and  1,342  kW 
for  30  minutes.  The  rating  category  for  this  power  plant  design  feature  is  a  measure  of 
merit  called  the  “maximum  continuous  drive”  for  the  platform.  Other  helicopters  may 
be  capable  of  achieving  even  greater  levels  of  sustained  performance  in  kilowatts,  but 
all  will  incorporate  the  maximum  continuous  drive  measure  of  merit.  Platforms  other 
than  helicopters  also  use  this  measure  of  merit,  such  as  the  Joint  Strike  Fighter  and  the 
Zumwalt-c\a.ss  Destroyer,  to  describe  the  power  plant  features.  There  are  considerable 
differences  between  the  technological  complexity  of  the  power  plant  design  required 
for  a  destroyer  and  that  of  a  jet  fighter,  but  the  common  measure  of  merit  defines  a  set 
of  similar  components.  By  this  definition,  a  measure  of  merit  includes  specific  techni¬ 
cal  components  as  well  as  systems  of  components  used  to  generate  a  particular  level 
of  performance,  such  as  the  maximum  continuous  drive.  This  broad  definition  was 
adopted  to  frame  the  decisionmaker’s  thinking  about  the  program  as  an  amalgamation 
of  processes  as  opposed  to  a  less  relevant  list  of  parts. 

We  crafted  a  graphical  display  to  help  the  decisionmaker  identify  the  most  impor¬ 
tant  features  through  the  selective  screening  process.  Figure  4.1  is  an  aggregate  image 
of  the  Longbow  Apache  measures  of  merit  as  ranked  for  level  of  system  complexity 
and  level  of  detail  required  to  access  each  feature’s  underlying  data  (the  level  of  detail 
increases  with  system  complexity).  The  specific  ranking  levels  for  each  axis  are  dis¬ 
cussed  later  in  this  chapter.  The  bubble  chart  depicts  the  frequency  of  measures  of 
merit  at  various  coordinates  on  the  matrix.  Larger  bubbles  indicate  a  larger  count  of 
measures  of  merit  at  the  particular  coordinate  than  do  smaller  bubbles.  The  different 
bubble  colors  simply  differentiate  one  coordinate  on  the  matrix  from  another.  Gener¬ 
ally,  measures  of  merit  that  rank  in  the  upper  right-hand  corner  of  the  chart — near 
the  upper  bound  of  each  complexity  and  level  of  detail  axis — contain  more  risk  than 
others.  In  a  root  cause  analysis,  the  reviewers  would  pay  extra  attention  to  the  history 
of  these  complex  programs  that  have  less  available  information. 

The  purpose  of  this  exercise  is  to  help  orient  a  decisionmaker  toward  components 
and  features  of  a  program  that  contain  the  most  risk  before  a  breach  occurs.  As  a  result 
of  this  exercise,  decisionmakers  will  be  more  able  to  focus  their  ongoing  analytic  efforts 
on  the  specific  critical  features  that  are  necessary  to  a  program’s  success.  The  graphical 
display  or  matrix  that  is  constructed  as  a  part  of  this  process  can  be  used  to  identify 
components  that  are  of  the  greatest  risk  to  a  program.  As  described,  this  is  done  by 


3  We  initially  assembled  a  list  of  measures  of  merit  used  in  Jane’s  All  the  World’s  Aircraft,  Jane’s  Ammunition 
Handbook,  Jane’s  Space  Systems  and  Industry,  and  Jane’s  Fighting  Ships  to  describe  the  different  programs. 
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Figure  4.1 
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relating  the  measures  of  merit  for  the  program  to  specific  levels  of  technological  com¬ 
plexity  and  levels  of  detail.  The  most  important  measures  of  merit  to  consider  are  those 
that  are  both  highly  complex  and  the  least  visible  (those  in  the  shaded  blue  square  in 
Figure  4.1  that  are  closer  to  the  upper  right-hand  corner  of  the  display).  For  a  detailed 
description  of  the  Longbow  Apache  risk  as  identified  by  the  matrix,  see  Figure  4.5  and 
the  associated  discussion  later  in  this  chapter. 

As  for  the  Longbow  Apache  and  the  other  programs  assigned  to  date,  we  coded 
each  measure  of  merit  for  both  complexity  and  level-of-detail  scales.  The  results  were  an 
alignment  of  the  rating  scales  to  a  coordinate  plane  in  a  two-dimensional  5x5  matrix. 
This  allows  the  assignment  of  every  Jane’s- based  measurement  to  a  coordinate,  as  seen 
in  Figure  4.1.  With  this  tool,  the  decisionmaker  or  reviewing  analyst  can  evaluate  the 
frequency  of  components  at  various  regions  of  the  resultant  “complexity-detail  matrix” 
to  get  a  better  view  of  the  measures  of  merit  that  contain  the  program  features  with  the 
most  potential  risk.  Construction  of  this  matrix  is  an  important  aspect  of  the  selective 
screening  thought  process.  Table  4.2,  below,  depicts  the  rating  scales  used  to  create  the 
matrix.  The  list  of  components  that  are  clustered  around  specific  regions  of  the  chart, 
such  as  at  the  upper  right-hand  corner,  should  be  evaluated  with  greater  scrutiny.  As 
an  example  of  the  codings  and  the  specific  measures  of  merit,  Table  4.1  displays  the 
data  associated  with  the  Longbow  Apache  gathered  from  Jane’s  All  the  World’s  Aircraft. 
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Table  4.1 

Longbow  Apache  Measures  of  Merit 


Design  Category 

Measures  of  Merit 

Level  of  Detail 
Required 

System 

Complexity 

Armament 

Areas 

1 

1 

Armament 

Maximum  ammunition  load 

1 

1 

Avionics 

Flight 

1 

1 

Capacity 

Crew 

1 

1 

Structure 

Dimensions,  external 

1 

1 

Structure 

Payload 

1 

1 

Armament 

Weights  and  loadings 

1 

3 

Armament 

Accuracy 

1 

5 

Performance 

Range 

1 

5 

Performance 

Speed 

1 

5 

Flying  controls 

Hover  hold 

2 

5 

Power  plant 

Crash/impact  resistance 

2 

5 

Power  plant 

Low  speed  stability 

2 

5 

Avionics 

Mission 

3 

3 

Avionics 

Radar 

3 

3 

Capacity 

Survivability 

3 

3 

Structure 

Crash/impact  resistance 

3 

3 

Systems 

Crash/impact  resistance 

3 

3 

Avionics 

Communications 

3 

4 

Avionics 

Instrumentation 

3 

4 

Flying  controls 

Fuselage  attitude 

3 

4 

Power  plant 

Continuous  maximum  drive 

3 

4 

Avionics 

Self-defense 

3 

5 

Avionics 

Target  acquisition 

4 

5 

Landing  gear 

Energy  absorption 

5 

1 

Power  plant 

Power  management 

5 

1 

Systems 

De-icing 

5 

1 

Systems 

Power  management 

5 

1 

Structure 

Rotor  noise  reduction 

5 

3 
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In  Table  4.2,  system  “complexity”  is  designed  as  an  indication  of  potential  design, 
engineering,  and  integration  difficulty  related  to  a  component.  The  rating  scale  of  com¬ 
plexity  range,  from  1  to  5  or  “straightforward”  to  “technologically  unproven.”  The 
definition  of  complexity  used  here  is  not  engineering,  operational,  or  even  program- 
atic  complexity  but  rather  a  broad  level  of  complexity  that  is  developed  through  the 
exercise.  For  example,  many  aspects  of  the  Joint  Strike  Fighter  were  technologically 
unproven — the  highest  order  of  complexity;  these  components  include  the  high-rate 
transonic  turning  flight  controls,  which  contributed  to  system  design  challenges  cited 
by  previous  RAND  research.4  At  the  other  end  of  the  complexity  spectrum  was  man¬ 
agement  of  the  landing  gear  for  the  Joint  Strike  Fighter.  Landing  gear  represents  a 
more  straightforward,  possibly  off-the-shelf  component  with  well-documented  tech¬ 
nology  that  would  have  contributed  very  little  to  the  overall  technical  risk. 

Table  4.2  also  defines  level  of  detail  or  visibility  as  an  indicator  of  the  amount 
of  effort  required  to  attain  suitable  documentation  on  the  component  of  interest.5 
This  axis  captures  the  extent  to  which  information  about  the  program  is  “pushed”  to 
the  decisionmaker.  From  the  experience  of  collecting  the  bibliographic  data,  we  con¬ 
structed  a  scale  that  ranges  from  1  to  5,  or  “advertised,”  meaning  that  documentation 
is  prevalent  and  pushed  to  decisionmakers,  and,  at  the  other  end,  the  information  is 
“buried,”  in  which  case  the  data  are  available  only  in  obscure  locations  that  are  also  dif¬ 
ficult  to  access.  Material  that  is  buried  includes  those  items  in  a  programs  bibliographic 
data  that  we  came  across  on  site  visits  or  found  almost  by  accident.  Support  from 
PARCA,  the  program  office,  and  other  gatekeeper  organizations  is  necessary  to  access 
and  to  even  learn  about  the  less  visible  data  that  seem  to  belong  in  the  buried  category, 
whereas  “advertised”  material  is  often  available  to  the  general  public.  See  Figure  4.2  for 


Table  4.2 

Complexity-Detail  Matrix  Ratings 


Value 

Level  of  Detail 
Required 

System  Complexity 

1 

Advertised 

Straightforward 

2 

Available 

Integrate  existing 

3 

Accessible 

Innovative  technique 

4 

Limited 

Complex  behavior 

5 

Buried 

Technologically  unproven 

4  See  Blickstein  et  al.,  2011,  p.  90ff,  for  RAND’s  root  cause  analysis  of  the  Joint  Strike  Fighter. 

5  The  challenges  associated  with  gaining  access  to  the  data  required  for  a  root  cause  analysis  is  discussed  in 
Chapter  Four  as  well  as  in  Appendix  G  of  Blickstein  et  al.,  forthcoming. 
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Figure  4.2 

Snapshot  of  a  Comparison  of  Program  Measurement  Systems 
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Each  program  is  characterized  by  a  sequence  of 
relevant  units  of  measure.  Precision,  speed,  and 
range  are  important  performance  measures  for 
the  Excalibur,  but  not  for  the  WGS. 
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The  performance  categories  used 
to  describe  the  Excalibur  are 
similar  to  those  used  for  the 
Longbow  Apache  and  the  Joint 
Strike  Fighter.... 


However,  the  power  plant  and 
structure  categories  are  very 
different. 


-  Performance 

2 

3 

2 

2 

4 

Precision 

i 
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1 

1 

1 

1 

4 

Speed 

1 

1 

1 

1 

4 

-  Power  Plant 
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2 

4 

Continuous  maximum  drive 

1 

1 

1 

3 

Crash/impact  resistance 

1 

1 

Downward  thrust  deflection 

►  1 

1 

Low  speed  stability 

1 

1 

2 

Power  management 

1 

1 

1 

3 

Solar  arrays  1 

1 

-  Structure 

4 

1 

2  1 

9 

5 

Beam 

1 

1 

Crash/impact  resistance 

1 

1 

2 

Dimensions.  External 

1 

1 

2 

NOTE:  The  bottom  of  the  figure  has  been  cut  off  for  display  purposes,  and  some  numbers  do  not  appear 
on  the  chart. 
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the  full  rating  scales  used  to  classify  the  list  of  measures  of  merit  that  was  assembled 
from  Jane’s  publications. 

Classification  of  the  measures  of  merit  was  based  on  RAND  experience  and 
familiarity  with  the  collection  process  and  the  intricacies  of  these  various  systems. 
We  reviewed  past  work  and  consulted  with  several  researchers  who  had  been  deeply 
engaged  in  the  initial  root  cause  analysis  of  each  platform  to  complete  the  exercise  and 
construct  the  specific  ratings  described  in  this  chapter. 


Comparing  Measures  of  Merit  Across  Programs 

In  broad  terms,  all  programs  have  measures  of  merit,  such  as  those  used  in  Jane’s  publi¬ 
cations,  and  some  of  these  measures  are  common  and  some,  such  as  those  that  describe 
the  Excalibur,  are  more  system-specific  than  those  found  in  major  platform  programs. 
For  example,  the  power  plant  “continuous  maximum  drive”  and  “low  speed  stability” 
of  the  Longbow  Apache  are  measures  of  merit  common  with  those  of  the  Joint  Strike 
Fighter.  Similarly,  performance  is  measured  in  terms  of  range  and  speed  for  the  Long¬ 
bow  Apache,  Joint  Strike  Fighter,  Excalibur,  and  the  DDG-1000.  The  performance  of 
Excalibur,  unlike  the  other  programs,  is  also  based  on  units  of  precision  or  accuracy. 
This  additional  unit  is  not  surprising,  as  Excalibur  was  built,  advertised,  and  is  required 
to  be  precise  so  as  to  reduce  collateral  damage,  particularly  in  urban  environments. 
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Figure  4.2  provides  a  comparative  display  of  power  plant  and  structural  measures 
of  merit  that  address  five  of  the  MDAPs  assigned  to  date  to  RAND  for  analysis.6  The 
columns  show  the  systems  and  the  rows  the  components.  The  rows  in  boldface  con¬ 
tain  the  sum  of  the  subcategories.  For  example,  the  Longbow  Apache  has  a  “4”  under 
power  plant,  which  is  the  total  of  four  subcategories:  continuous  maximum  drive, 
crash  impact/resistance,  low  speed  stability,  and  power  management.  As  the  figure 
shows,  the  Joint  Strike  Fighter  has  similar  systems,  which  would  enable  a  comparison 
across  systems. 

Although  similar  measures  of  merit  might  be  used  to  describe  different  programs, 
the  calculations  are  based  on  integration  of  the  specific  characteristics  of  very  different 
technologies.  For  example,  structural  “crash/impact  resistance”  means  the  amount  of 
damage  (by  caliber  round)  that  the  airframe  and  fuel  cells  of  the  Longbow  Apache  can 
sustain.  On  the  other  hand,  the  same  Jane’s- listed  measure  for  the  DDG-1000  refers  to 
the  wide  distribution  of  ballistic  impact  across  the  hull  to  reduce  the  risk  of  single-hit 
ship  loss.  The  fuel  cells  and  air  frame  designed  for  the  Longbow  Apache  are  relatively 
standard  across  the  light  attack  helicopter  industry;  whereas,  optimized  for  stealth,  the 
DDG-1000  hull  design  is  the  result  of  more  innovative  engineering.  In  each  case,  the 
single  measure  of  merit  reflects  different  levels  of  system  complexity.  The  difficulty  in 
having  measures  that  are  useful  and  actionable  is  that  if  the  category  is  homogenized 
enough,  the  meaning  is  lost. 

Because  of  this  need  for  a  more  tailored  understanding  of  the  measures  of  merit, 
RAND  developed  the  selective  screening  exercise.  A  product  of  the  exercise  is  the 
complexity-detail  matrix  addressed  above  (Table  4.2)  and  found  in  Figures  4.5  through 
4.9,  below,  for  each  program  assigned  to  date  to  depict  the  programs  through  the  asso¬ 
ciated  “complexity”  and  “level  of  detail”  or  visibility  ratings.  The  complexity-detail 
matrix  shows  how  frequently  program  components  exhibit  various  levels  of  complexity 
and  visibility  through  the  use  of  larger  and  smaller  bubbles.  For  better  understanding, 
some  of  the  bubbles  in  Figures  4.5  through  4.9  have  the  specific  program  component 
nomenclature  indicated.  This  mechanism  is  an  example  of  how  existing  data  can  help 
identify  underlying  complexities.  Figures  4.3  and  4.4  depict  the  complexity  and  level  of 
detail  axis  on  the  matrix,  using  the  scales  from  Table  4.2.  A  greater  explication  of  the 
axes  and  their  use  is  discussed  in  the  program-specific  example  matrices. 

The  program  components  that  are  more  complex  and  also  less  visible  carry  much 
of  the  risk  related  to  program  failure.  These  critical  features  to  a  program  can  be  iden¬ 
tified  by  reviewing  the  components  that  are  within  specific  coordinate  clusters  on  the 
matrix.  The  following  section  provides  specific  program  examples  using  the  matrix. 
The  approach  followed  is  somewhat  akin  to  the  methodology  used  to  analyze  a  quad 


6  All  measures  used  in  the  analysis  in  this  chapter  are  entirely  based  on  the  most  recent  program  descriptions 
from  Jane’s  All  the  World’s  Aircraft,  Jane’s  Ammunition  Handbook,  Jane’s  Space  Systems  and  Industry ,  and  Jane’s 
Fighting  Ships. 
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Figure  4.3 

System  Complexity  Analysis 
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Figure  4.4 
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chart  in  that  the  displays  focus  the  observer  on  specific  data  points  within  particular 
regions  of  the  depiction. 


Program-Specific  Examples  for  Using  the  Exercise  for  Complexity- 
Detail  Matrix  Through  the  Selective  Screening  of  Critical  Components 

The  following  charts  are  based  on  several  of  the  programs  assigned  to  RAND  to  date. 
In  each,  the  scales  used  on  the  horizontal  and  vertical  axes  are  those  for  complexity  and 
level  of  detail  or  visibility  that  are  depicted  in  Figures  4.2,  4.3,  and  4.4.  The  root  cause 
analysis  material  for  Excalibur  is  from  the  root  cause  analysis  in  this  report,  whereas 
the  Longbow  Apache,  Joint  Strike  Fighter,  DDGT000,  and  Wideband  Global  Satel¬ 
lite  are  from  the  material  generated  for  the  associated  root  cause  analyses  detailed  in 
Blickstein  et  al.,  2011. 

Longbow  Apache 

By  Army  accounts,  the  Longbow  Apache  incorporated  15  cutting-edge  technologies; 
this  magnitude  of  complexity  introduced  greater  integration  and  development  chal¬ 
lenges  to  the  program.  Figure  4.5  depicts  the  Longbow  Apache  measures  of  merit  on 
the  matrix.  The  chart  relates  the  various  levels  of  complexity  to  level  of  detail  or  vis¬ 
ibility  of  the  underlying  components.  The  feature  clusters  that  are  in  the  shaded  blue 

Figure  4.5 
Longbow  Apache 
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square  reflect  those  of  the  greatest  potential  risk  because  they  are  both  complex  and  the 
information  documenting  the  components  are  more  buried  than  in  other  systems.  The 
components  within  the  blue  shaded  square  region  of  the  chart  include  the  rotor  noise 
reduction,  continuous  maximum  drive  of  the  power  plant,  and  avionics  and  crash/ 
survivability  in  the  structure  design.  The  RAND  root  cause  analysis  of  the  Longbow 
Apache  revealed  that  the  increase  in  costs  associated  with  the  rotor  and  the  power  plant 
drive  system,  which  was  documented  at  the  development  stage  MS  B,  contributed  to 
the  Nunn-McCurdy  breach. 

Although  not  all  of  the  components  that  fall  within  the  blue  shaded  region  con¬ 
tributed  to  the  root  cause  of  the  breach,  knowing  the  few  critical  features  to  follow  is 
beneficial  to  the  analyst. 

F-35  Lightning  II  Joint  Strike  Fighter 

For  the  F-35  Lightning  II  Joint  Strike  Fighter,  the  most  complex  systems  were  also 
difficult  to  integrate  and  led  to  delays.  A  paramount  need  for  stealth  in  the  new  Joint 
Strike  Fighter  required  an  innovative  flight  avionics  arrangement,  a  unique  trapezoi¬ 
dal  mid-wing  configuration,  and  internal  weapon  bays.  The  need  for  stealth,  with  the 
design  of  more  technologies  inside  the  airframe,  resulted  in  an  excess  of  weight  in 
the  F-35B  and  F-35C  variants  that  was  difficult  for  the  program  to  shed.  Appropriate 
weight  growth  given  the  requirements  was  also  not  accounted  for  in  the  MS  B  baseline 
2001  cost  estimates,  as  these  were  based  on  legacy  aircraft  data.  Redesigns  to  reduce 
weight  contributed  to,  but  were  not  the  sole  cause  that  increased  the  unit  cost  beyond 
the  approved  levels. 

Information  about  the  Joint  Strike  Fighter  weight  distribution  and  trimming 
challenges  associated  with  the  need  for  stealth  was  found  in  a  PowerPoint  presentation 
provided  by  the  Lockheed  Martin  System  Engineering  Director  on  April  23,  2010. 
This  information  was  considered  “buried”  on  the  level-of-detail  scale  as  it  was  uncov¬ 
ered  only  while  joining  the  PARCA  office  on  a  site  visit  to  Lockheed  Martin.  So  buried 
information  can  be  well  known;  the  weight  problems  were  reported  widely.  Several  of 
the  slides  prepared  for  the  PARCA  office  by  Lockheed  Martin  characterized  the  weight 
growth  as  a  considerable  barrier  over  a  ten-year  time  line.7  This  information  was  dif¬ 
ficult  to  acquire  but  became  available  through  the  support  of  the  PARCA  office. 

Figure  4.6  captures  the  Joint  Strike  Fighter  measures  of  merit  as  described  by 
complexity  and  component  visibility.  The  shaded  region  in  the  upper  right-hand  corner 
identifies  components  that  exhibit  both  a  high  level  of  complexity  and  a  low  level  of 
visibility.  Examined  further,  the  cluster  includes  components  such  as  the  flying  con¬ 
trols  for  high-rate  transonic  flight,  the  thermal  management  system,  and  the  various 
avionics  and  power  plant  systems.  As  described  in  the  Joint  Strike  Fighter  root  cause 


7  Paul  Park,  System  Engineering  Director,  “PARCA  Review:  Air  Vehicle,  F-35  Lightning  II,”  Lockheed  Martin, 
PowerPoint  reference  file:  “AVT_Park_Rev02.ppt,”  April  23,  2010. 
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Figure  4.6 
Joint  Strike  Fighter 
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analysis,  the  integration  of  these  complex  components  into  a  stealth  design  resulted  in 
an  increase  to  the  overall  weight,  which  was  not  accounted  for  in  the  MS  B  baseline. 

The  underlying  data  associated  with  Figure  4.6  are  depicted  in  Table  4.3. 

Components  in  the  shaded  region  of  Figure  4.6  contributed  to  the  Nunn- 
McCurdy  breach  because  the  process  of  designing  these  complex  technologies  into  the 
stealth  frame  resulted  in  a  rapid  increase  in  overall  platform  weight.  Knowing  the  list 
of  components  to  focus  on  would  lead  the  analyst  toward  overall  underlying  trends 
that  contributed  to  risk  in  the  program. 

Zumwalt-Oass  Destroyer  (DDG-1000) 

Similar  to  the  Joint  Strike  Fighter,  the  Zumwalt-class  Destroyer  (DDG-1000)  is 
designed  for  stealth  and  integrates  advanced  technologies  but  in  this  case  for  long- 
range  land  attacks.  Regardless  of  the  design  and  integration  challenges  associated 
with  complex  systems,  the  RAND  root  cause  analysis  detailed  in  the  PARCA  I  report 
(Blickstein  et  ah,  2011)  identified  that  the  quantity  change  from  ten  to  three  ships  was 
the  main  driver  of  the  unit  cost  growth  that  resulted  in  the  breach.  More  important, 
the  RAND  research  found  that  another  cause  of  the  breach  may  have  been  an  unreal¬ 
istic  assessment  of  the  baseline  at  MS  B.  In  the  case  of  DDG-1000,  Figure  4.7  could 
have  been  used  by  an  analyst  to  identify  the  levels  of  complexity  that  differentiate  the 
DDG-1000  from  the  DDG-51  before  assessing  the  baseline  cost. 
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Table  4.3 

Joint  Strike  Fighter  Measures  of  Merit 


Design  Category 

Level  of  Detail  System 
Measures  of  Merit  Required  Complexity 

Armament 

Length 

1 

1 

Armament 

Payload 

1 

1 

Armament 

Weight  and  loadings 

1 

2 

Structure 

Penetration 

1 

1 

Flying  controls 

Lifting  glide  ratio 

1 

3 

Flying  controls 

Stabilization 

1 

3 

Performance 

Precision 

1 

5 

Performance 

Range 

1 

5 

Performance 

Speed 

1 

5 

Systems 

Height  of  burst 

2 

3 

Systems 

Guidance 

2 

5 

Systems 

Launch  control 

3 

2 

Systems 

Sensors 

3 

2 

Figure  4.7 
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The  DDG-51  is  a  multimission  destroyer,  designed  for  air  defense  and  ocean- 
based  operations,  whereas  the  DDG-1000  is  designed  for  operations  in  littoral  areas 
and  naval  surface  fire  support.  To  support  this  fundamental  difference  in  opera¬ 
tional  need,  the  DDG-1000  required  the  integration  of  several  new  technologies.  The 
cost  structure  for  the  DDG-51  was  used  as  a  basis  for  the  DDG-1000  initial  pric¬ 
ing,  although  the  two  have  very  different  platforms.  The  blue  shaded  region  in  Figure 
4.7  captures  several  of  the  technologies  that  differentiated  the  DDG-1000  from  the 
DDG-51.  To  accommodate  the  new  technologies,  the  DDG-1000  is  structurally  larger 
and  has  more  automation  features  allowing  it  to  operate  with  fewer  crewmembers  (142 
sailors  compared  with  approximately  300  on  the  Navy’s  Arleigh  Burke— class  Destroy¬ 
ers  [DDG-51]  and  Ticonderoga- class  Cruisers  [CG-47]).  The  blue  shaded  region  in 
Figure  4.7  captures  the  increased  draft  and  other  structural  differences  as  well  as  the 
integrated  electric  drive  propulsion  system  and  automation  technologies. 

These  differences  can  also  be  viewed  through  the  underlying  data  depicted  in 
Table  4.4. 

In  the  case  of  the  DDG-1000,  the  analyst’s  early  understanding  of  the  technical 
complexities  that  contribute  to  risk  in  the  system  may  have  helped  to  better  specify  the 
baseline  costs.  This  type  of  evaluation  of  the  underlying  features  and  critical  compo¬ 
nents  of  a  program  is  valuable  not  only  in  the  root  cause  analysis  but  also  in  the  general 
characterization  of  a  program  before  it  is  at  risk  of  failure. 

Excalibur 

The  Excalibur  projectile  program  was  merged  with  the  Trajectory  Correctable  Muni¬ 
tions  program  in  2002  to  complement  the  Future  Combat  System  NLOS  Launch 
System  (NLOS-LS)  launch  capability.  The  NLOS-LS  was  ultimately  canceled  in  May 
2010,  and  the  Excalibur  Nunn-McCurdy  breach  was  announced  in  August  2010.  The 
RAND  root  cause  analysis  in  this  report  points  to  inaccurate  initial  estimates  of  cost 
and  a  change  to  the  operational  concept  of  the  device  as  the  primary  causes,  but  the 
analysis  also  identified  several  long-term  technology  challenges.  Similar  to  the  DDG- 
1000,  the  design  and  development  history  of  Excalibur  suggests  that  earlier  identifica¬ 
tion  of  the  critical  components  and  potential  areas  of  risk  could  have  helped  to  avert 
the  major  challenges  that  contributed  to  increased  costs. 

The  shaded  blue  regions  of  Figure  4.8  capture  the  technologies  that  represented 
the  most  risk  for  the  Excalibur  system.  Described  in  greater  detail  in  Chapter  Five  of 
this  report,  these  include  the  systems  components  for  the  height  of  burst,  guidance, 
launch  control,  and  sensors.  These  system  components  were  necessary  for  the  high  level 
of  precision  required  for  Excalibur,  but  early  failures  in  these  systems  led  to  long-term 
scheduling  delays  and  previously  unforeseen  costs  that  contributed  to  the  overall  pro¬ 
gram  risk. 

The  underlying  data  for  Figure  4.8  are  depicted  in  Table  4.5. 
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Table  4.4 

Zumwalt-C\ass  Destroyer  Measures  of  Merit 


Design  Category 

Measures  of  Merit 

Level  of  Detail 
Required 

System 

Complexity 

Armament 

Guns 

1 

1 

Armament 

Missiles 

1 

1 

Armament 

Torpedoes 

1 

1 

Capacity 

Crew 

1 

1 

Equipment 

Decoys 

1 

1 

Equipment 

Unmanned  aerial  vehicles/helicopters 

1 

1 

Performance 

Range 

1 

1 

Performance 

Speed 

1 

1 

Structure 

Payload 

1 

1 

Structure 

Beam 

1 

4 

Structure 

Displacement 

2 

3 

Structure 

Draft 

2 

3 

Structure 

Length 

2 

3 

Structure 

Survivability 

2 

3 

Structure 

Crash/impact  resistance 

2 

4 

Structure 

Stealth  optimization 

2 

4 

Systems 

Radars:  air/surface  search 

3 

3 

Systems 

Radars:  navigation 

3 

3 

Systems 

Radars:  surface  search 

3 

3 

Systems 

Sonars:  active  search 

3 

3 

Systems 

Sonars:  attack  search 

3 

3 

Systems 

Sonars:  passive  array 

3 

3 

Power  plant 

Continuous  maximum  drive 

3 

4 

Power  plant 

Power  management 

5 

2 

Structure 

Power  management 

5 

1 
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Figure  4.8 
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Table  4.5 

Excalibur  Projectile  Measures  of  Merit 

4 

5 

Design  Category 

Level  of  Detail  System 
Measures  of  Merit  Required  Complexity 

Armament 

Length 

1 

1 

Armament 

Payload 

1 

1 

Armament 

Weight  and  loadings 

1 

2 

Structure 

Penetration 

1 

1 

Flying  controls 

Lifting  glide  ratio 

1 

3 

Flying  controls 

Stabilization 

1 

3 

Performance 

Precision 

1 

5 

Performance 

Range 

1 

5 

Performance 

Speed 

1 

5 

Systems 

Height  of  burst 

2 

3 

Systems 

Guidance 

2 

5 

Systems 

Launch  control 

3 

2 

Systems 

Sensors 

3 

2 
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The  information  gained  from  analysis  of  the  complexity  and  visibility  of  various 
components  of  the  Excalibur  would  help  the  analyst  identify  the  critical  features  that 
should  be  closely  monitored  during  the  design,  development,  integration,  and  produc¬ 
tion  phases  of  the  program.  This  example  suggests  that  alternative  views  for  these  com¬ 
plex  weapons  programs  are  necessary  to  better  understand  the  nature  and  potential 
sources  of  risk  throughout  the  program’s  life  cycle. 

Wideband  Global  Satellite 

Fast  and  clear  communication  between  the  battlefront,  mission  headquarters,  and 
defense  strategic  commands  is  critical  to  deploying  resources.  Wideband  Global  Satel¬ 
lite  is  a  component  of  the  military’s  investment  strategy  to  build  the  communications 
infrastructure  for  the  future.  Unlike  other  programs,  the  WGS  cost  structure  placed 
more  of  the  burden  on  the  program  engineering  prime  contractor,  obligating  Boeing 
to  recoup  development  costs  from  the  commercial  market  rather  than  fully  supporting 
the  overhead  costs  through  funding  from  DoD. 

Over  time,  the  WGS  evolved  from  a  commercial  platform  with  military  features 
to  an  almost  entirely  military-purpose  device.  Identified  by  the  RAND  root  cause 
analysis,  a  growing  difference  between  the  commercial  and  military  variants  of  the 
WGS  resulted  in  components  that  were  maintained  through  successive  designs  to  meet 
the  military  requirements  that  were  no  longer  “commercial  off-the-shelf.”  The  com¬ 
mercial  market  took  note  of  this  evolution,  and  the  commercial  WGS  ratings  declined 

Figure  4.9 
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as  a  result.  The  WGS  Nunn-McCurdy  breach  was  caused  by  internal  cost  growth  and 
external  market  conditions. 

The  shaded  blue  regions  of  Figure  4.9  identify  several  of  the  more  technically 
complex  features  of  the  WGS.  These  include  systems  components  such  as  the  digital 
channelizer,  filter  and  route  bandwidth  routines,  cross-banded  communications,  and 
the  data  transmission  rate.  As  described  in  the  RAND  root  cause  analysis,  these  tech¬ 
nologies  are  at  the  heart  of  the  differences  between  the  commercial  and  military  vari¬ 
ants.  Boeing  continued  to  support  the  high-power  1TS7021TP  bus  for  rapid  data  trans¬ 
mission  rates  for  the  military  WGS  but  switched  to  the  medium-power  1TS702MP  bus 
for  the  commercial  version.  Boeing  maintained  transponder  and  channelizer  support 
for  military  data  bands  unnecessary  to  the  commercial  market — the  X-band  is  used  for 
older  military  satellites  and  communications  systems,  and,  although  the  Ka-band  has 
been  explored  for  commercial  purposes,  markets  have  trended  toward  fiber  optic  and 
cell  phone  data  handlers  rather  than  satellite-based  communications. 

The  underlying  data  for  Figure  4.9  are  captured  in  Table  4.6. 

As  with  the  DDG-1000  example,  understanding  the  technological  differences 
between  somewhat  similar  platforms — the  DDG-1000  versus  the  DDG-51  and  the 
WGS  commercial  versus  the  WGS  military — would  provide  the  analyst  with  a  clearer 
perspective  of  the  nuanced  points  of  risk  in  these  complex  systems.  In  the  case  of  the 
WGS,  this  information  was  critical  early  in  the  initial  pricing  as  well  as  through  the 
WGS  design  cycle  as  the  commercial  market  conditions  changed. 


Table  4.6 

Wideband  Global  Satellite  Measures  of  Merit 


Design  Category 

Measures  of  Merit 

Level  of  Detail 
Required 

System 

Complexity 

Design  life 

Years 

1 

3 

Structure 

Payload 

1 

1 

Systems 

Antennas 

1 

1 

Systems 

Launch  control 

1 

2 

Orbits 

Distance 

1 

3 

Power  plant 

Solar  arrays 

1 

3 

Systems 

Communications  payload 

1 

5 

Systems 

Digital  channelizer 

2 

3 

Systems 

Filter  and  route  bandwidth  routines 

2 

3 

Systems 

Cross-banded  communications 

2 

4 

Systems 

Data  transmission  rate 

2 

4 

Power  plant 

Power  management 

5 

1 
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The  15  percent  risk  premium  accounting  artifact  that  was  associated  with  the 
Block  II  units  and  identified  as  a  considerable  cause  of  the  WGS  failure  would  not  have 
been  identified  by  the  matrix  approach.  Although  the  matrix  is  a  valuable  tool  for  iden¬ 
tifying  the  technical  components  that  contribute  risk  to  the  programs,  we  recommend 
that  the  program  analyst  and  decisionmaker  view  the  programs  with  a  variety  of  tools 
to  gain  a  more  robust  understanding  of  the  program  risk. 


Conclusion 

MDAPs  are  complex  systems  designed  to  support  specific  requirements.  As  the  needs 
of  the  battlefield  evolve,  so  will  the  demand  for  integrated,  better,  and  faster  technolo¬ 
gies.  The  programs  described  in  this  report  and  reviewed  in  this  chapter  were  designed 
to  integrate  cutting-edge  components,  and  in  many  cases  that  requirement  contributed 
to  the  overall  program  risk.  Although  each  program’s  life  cycle  story  was  slightly  dif¬ 
ferent,  a  common  theme  is  that  the  additional  attention  to  a  selection  of  critical  com¬ 
ponents  early  in  a  program’s  development  could  help  identify  underlying  risk  before  a 
program  fails  or  a  Nunn-McCurdy  breach  root  cause  analysis  is  required. 

This  chapter  describes  just  one  exercise  that  could  be  adopted  to  help  identify 
the  most  critical  components  from  the  long  list  of  program  features  required  of  these 
complex  systems.  Experience  in  analyzing  the  root  cause  reasons  for  a  Nunn-McCurdy 
breach  suggests  that  focus  on  these  critical  features  earlier  will  help  the  decisionmaker 
or  reviewing  analyst  to  identify  underlying  patterns,  and  this  understanding  would 
help  identify  the  initiating  hypothesis  discussed  in  this  report.  Chapter  Five  furthers 
this  discussion  of  critical  component  risk  with  an  analysis  of  the  technical  component 
risk  in  the  Excalibur  example.  By  focusing  on  the  critical  components  in  Excalibur, 
analysts  would  have  been  able  to  trace  scheduling  delays,  software  failures,  and  simu¬ 
lation  and  test  failures  through  a  two-year  and  seven-month  period  of  the  programs 
technical  component  history  well  before  the  Nunn-McCurdy  breach  occurred.  This 
list  of  critical  components  was  developed  as  a  result  of  the  selective  screening  thought 
process  described  by  this  chapter. 

Although  there  can  be  a  variety  of  methods  for  examining  a  defense  program, 
more  work  needs  to  be  done  to  better  characterize  programs  by  their  critical  compo¬ 
nents.  These  important  features  are  often  the  most  difficult  to  design,  integrate,  and 
develop  and  therefore  carry  much  of  the  overall  program  risk. 


CHAPTER  FIVE 


Assessment  of  Technical  Risk  in  Weapons  Programs 


Defense  acquisition  programs  are  generally  large  and  often  require  the  complex  inte¬ 
gration  of  many  technologies.  This  chapter  outlines  a  way  to  make  the  complexity  of 
a  defense  program  more  transparent  for  the  decisionmaker  or  reviewing  analyst  by 
examining  a  manageable  list  of  components  organized  by  complexity  and  the  level  of 
detail  required  to  study  the  component.  This  chapter  documents  a  process  to  aid  in  the 
early  identification  of  technical  risk  in  the  most  critical  components. 

As  a  result  of  the  RAND  root  cause  analysis  of  Excalibur,  several  questions 
related  to  the  programs  technical  problems  emerged.  Using  a  selective  screening  of 
critical  components  process,  outlined  in  this  chapter,  the  components  most  necessary 
to  the  success  of  the  program  were  identified.  The  following  is  an  exploration  into  the 
development  history  of  those  critical  components  based  on  a  series  of  Defense  Contract 
Management  Agency  (DCMA)  parts  management  program  (PMP)  and  DAES  risk 
assessments.  We  track  those  components  to  see  if  either  DCMA  or  DAES  would  have 
presaged  the  problems. 

The  Excalibur  DCMA  PMP  risk  assessments  were  discovered  within  a  series 
of  monthly  program  reports  that  were  originally  produced  by  the  DCMA  program 
integrator  and  directed  to  the  project  manager,  combat  ammo  systems.  The  monthly 
reports  update  cost  performance  data  for  the  period,  current  business  operations,  and 
technical  component-level  risk  to  the  system.  Two  measures  of  DCMA  risk  are  used 
throughout  the  report  series:  the  IMU  and  the  GPS.  The  GPS  and  IMU  were  identi¬ 
fied  through  the  selective  screening  of  critical  components  process  as  key  aspects  of  the 
primary  goals  of  the  platform. 

Section-level  measures  of  risk  are  based  on  the  DAES  risk  tri-color  rating  scale, 
which  is  explained  in  the  Department  of  Defense  “Risk  Management  Guide  to  DoD 
Acquisition.”1  Technical  component-level  risk  is  captured  using  the  DCMA  tri-color 


1  Department  of  Defense,  “Risk  Management  Guide  to  DoD  Acquisitions,”  Version  6,  Washington,  D.C., 
August  2006. 
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rating  scale  (green,  yellow,  and  red),  which  is  based  on  a  risk  analysis  of  the  probability 
and  consequence  of  failure  to  meet  the  program  requirements.2 

For  the  purpose  of  this  effort,  fluctuations  in  both  the  DAES  section-level  and 
the  DCMA  technical  component-level  risk  ratings  were  used  to  identify  periods  of 
interest  in  the  Excalibur  time  line.  Through  this  work,  we  identified  a  pattern  of  indi¬ 
vidual  system  shocks  and  serial  problems  that  contributed  to  larger  and  longer-term 
complications  for  the  platform.  Future  problems  for  similar  programs  might  be  better 
anticipated  by  using  monthly  measures,  such  as  the  DCMA  technical-level  risk  assess¬ 
ment,  to  earlier  recognize  emerging  risks  that  pose  hurdles  to  meeting  the  program 
requirements. 


Background 

In  its  two-decade  history,  Excalibur  underwent  changes  in  design,  decreases  in  the 
quantity  planned,  and  two  related  program  cancellations.  Presented  in  Chapter  Two, 
the  RAND  root  cause  analysis  of  Excalibur  determined  that  the  most  significant  con¬ 
tributors  to  the  Nunn-McCurdy  breach,  announced  during  August  2010,  were  inac¬ 
curate  estimates  of  the  preliminary  unit  cost  as  well  as  a  change  to  the  operational  con¬ 
cept  of  the  unit.  For  a  program  such  as  Excalibur  that  had  evolved  from  an  unguided 
projectile  to  an  entirely  different  full-GPS  and  IMU-guided  device,  these  findings  are 
not  unexpected. 

The  root  cause  analysis  also  ruled  out  other  difficulties  that  the  program  had  as 
nonfactors  in  the  decision  to  reduce  the  quantity  planned  that  triggered  the  initial 
breach.  Many  of  these  were  technical  problems  that  were  realized  during  developmen¬ 
tal  testing,  in  the  production  process,  and  as  a  result  of  the  relocation  of  contractor 
facilities.  Although  these  challenges  were  not  identified  as  prime  factors  leading  to  the 
Nunn-McCurdy  breach,  they  did  delay  the  program  schedule  and  increase  the  unit  as 
well  as  overall  costs.3  The  questions  that  emerged  from  these  technical  considerations 
include:  How  did  these  problems  with  technical  aspects  evolve  from  early  warnings 
to  larger  challenges?  And  what  can  be  done  to  identify  potential  longer-term  risk  to 
weapons  programs? 


2  Defense  Contract  Management  Agency,  Guide  Book:  Parts  Management  Program — Risk  Planning ,  Washing¬ 
ton  D.C. 

3  The  GAO  annual  report,  Defense  Acquisitions:  Assessments  of  Selected  Weapons  Programs,  has  featured  the  Excal¬ 
ibur  program  in  several  editions.  The  most  recent  have  concluded  that  the  program’s  history  of  technical,  require¬ 
ment,  and  other  changes  have  led  to  unexpected  costs,  schedule  delays,  and  reductions  in  planned  procurement. 
For  more  information,  see  GAO,  2010,  2006,  2007,  2008,  and  2009,  GAO-09-326SP,  GAO-08-467SP. 
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History  of  Excalibur  Unit  Costs 

Since  2004,  the  GAO  has  tracked  the  challenges  facing  major  weapon  programs  in  the 
Department  of  Defense  and  each  year  has  included  a  review  of  Excalibur.  The  GAO 
found  existing  challenges,  before  2004,  that  led  to  the  Excalibur  program’s  situation. 
These  challenges  include  several  restructures  and  mergers  to  incorporate  other  pro¬ 
grams.  Depicted  in  Figure  5.1,  the  demand  for  the  projectile  fell  from  200,000  units 
to  only  6,294  between  1997  and  2010,  an  overall  reduction  in  demand  that  occurred 
during  four  time  periods:  2003,  2004,  2005,  and  2010.  Note  that  the  price  per  unit 
continued  to  increase  between  2005  and  2010. 

The  GAO  concluded  that  the  Excalibur  program’s  history  of  technical,  require¬ 
ments,  and  other  changes  led  to  schedule  delays  that  caused  unexpected  costs  and 
reductions  to  planned  procurement.  To  address  those  issues,  the  complexity  analysis 
documented  in  Chapter  Four  was  used  to  identify  the  most  complex  components  with 
the  least  visibility. 


Identifying  Areas  of  Potential  Risk 

Chapter  Four  detailed  a  process  for  identifying  areas  of  potential  program  risk  based 
on  a  RAND-developed  methodology  to  relate  measures  of  merit — the  standard  mea- 

Figure  5.1 
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sures  used  by  a  variety  of  Jane’s  publications4  to  describe  programs — to  the  complexity 
and  visibility  of  the  components  they  are  used  to  measure.  Relying  on  RAND  expe¬ 
rience  with  the  programs,  previous  work  with  the  source  documents  and  data,  and 
familiarity  with  the  underlying  components,  we  crafted  a  matrix  to  explore  the  rela¬ 
tionship  between  complexity  and  level  of  detail  or  visibility.  See  Table  5.1  for  the  full 
rating  scales  used  to  classify  the  measures  of  merit. 

We  coded  each  measure  of  merit  for  both  complexity  and  level-of-detail  scales. 
The  results  were  an  alignment  of  the  rating  scales  to  a  coordinate  plane  in  a  two  dimen¬ 
sional  5x5  matrix.  This  allows  the  assignment  of  every  Jane’s-  based  measurement  to  a 
coordinate.  With  this  tool,  the  decisionmaker  or  analyst  can  evaluate  the  frequency  of 
components  at  various  regions  of  the  resultant  complexity-detail  matrix  to  better  view 
components  of  potential  risk.  In  the  case  of  Excalibur,  as  displayed  in  Figure  5.2,  the 
most  interesting  measurements  (in  the  blue  shaded  square)  were  the  least  visible  and 
more  complex  system  components — the  systems  components  for  the  height  of  burst, 
guidance,  launch  control,  and  sensors. 

Based  on  the  complexity  and  level  of  detail  review,  the  measurements  identified 
describe  systems  components.  From  experience  working  with  the  source  documents 
for  Excalibur,  the  DCMA  monthly  progress  reports  were  the  appropriate  informa¬ 
tion  source  for  more  information  about  these  components.  Had  the  complexity-detail 
analysis  revealed  that  the  source  of  the  problem  was  the  projectile’s  aerodynamics  or 
canards,  then  the  best  source  of  data  may  have  been  the  engineering  and  material  test¬ 
ing  logs  or  external  research  and  design  studies. 

This  granular  review  of  the  Excalibur  program’s  technical  component-level  prog¬ 
ress  was  necessary  because  of  questions  raised  as  a  result  of  the  root  cause  analysis 
reported  in  Chapter  Two.  Additionally,  we  recognized  that  the  Excalibur  analysis 
required  different  considerations.  Analysts  found  that  the  most  detrimental  events  in 


Table  5.1 

Complexity-Detail  Matrix  Ratings 


Value 

Level  of  Detail 
Required 

System  Complexity 

1 

Advertised 

Straightforward 

2 

Available 

Integrate  existing 

3 

Accessible 

Innovative  technique 

4 

Limited 

Complex  behavior 

5 

Buried 

Technologically  unproven 

4  We  initially  assembled  a  list  of  measures  of  merit  used  in  jane’s  All  the  World’s  Aircraft,  Jane’s  Ammunition 
Handbook,  Jane’s  Space  Systems  and  Industry,  and  Jane’s  Fighting  Ships  to  describe  the  different  programs. 


Assessment  of  Technical  Risk  in  Weapons  Programs  67 


Figure  5.2 

Complexity-Detail  Matrix  for  the  Excalibur  Program 


Level  of  detail  required 
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Excalibur’s  development  history  were  inaccurate  estimates  of  the  preliminary  unit  cost 
as  well  as  a  change  to  the  operational  concept  of  the  unit.  However,  during  the  analy¬ 
sis,  RAND  reviewers  mentioned  technical  concerns  but  generally  with  limited  specific 
detail.  Late  in  the  analysis,  the  DCMA  monthly  progress  reports  were  transmitted  to 
RAND  almost  as  an  afterthought. 

The  newly  available  greater  level  of  technical  detail  provided  by  the  monthly  prog¬ 
ress  reports  allowed  for  a  more  in-depth  review  of  the  Excalibur  technical  concerns. 
This  level  of  detail  is  available  to  the  defense  acquisitions  community  but  was  made 
available  to  RAND  only  for  the  Excalibur  Nunn-McCurdy  root  cause  analysis. 


Source  Documents  Associated  with  the  Critical  Components 

The  DCMA  monthly  reports  that  were  available  for  Excalibur  covered  a  two-year  and 
seven-month  period  of  the  projectile’s  13-year  history,  but  enough  data  were  avail¬ 
able  to  identify  patterns.  These  patterns  of  problems  in  the  programs  technical  history 
contributed  to  the  longer-term  risk  of  not  meeting  the  platform  requirements.  Early 
identification  of  these  patterns  is  necessary  to  avoid  longer-term  problems  that  push 
programs  toward  a  Nunn-McCurdy  breach,  which  occurs  when  a  system  exceeds  its 
initial  expected  costs.  Because  of  the  nature  of  the  problems  identified,  we  determined 
that  our  analytical  methodology  had  to  devolve  to  a  lower  level  of  detail. 
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We  extracted  the  desired  level  of  detail  information  from  existing  reports.  Figure 
5.3  is  an  image  from  the  monthly  program  reports.  Risk  rating  values  from  the  DCMA 
reports  were  compiled  to  reflect  detail  status  akin  to  that  in  Figure  5.6,  below. 

We  found  that,  over  all  reporting  periods,  the  overall  rating  for  the  Excalibur 
program  never  changed  from  yellow  (potential  or  actual  problems)  despite  a  number 
of  underlying  technical  problems.  Figure  5.4  depicts  changes  to  the  Excalibur  DAES 
section-level  risk  ratings.  The  image  captures  section-level  deviations  from  the  yellow 
overall  rating. 

Note  that  the  1.2  business  section  remained  green  (advisory)  until  March  2010,  at 
which  point  it  changed  to  yellow  (the  same  as  the  overall  DAES  rating  for  that  period). 
The  technical  performance  rating  deviates  from  the  overall  DAES  rating  from  Decem¬ 
ber  2008  through  March  2009  as  depicted  by  the  red  (high  risk)  to  yellow  branch  at 
the  top  of  the  illustration.  Schedule  performance  fluctuates  from  yellow  to  green  to 
yellow  between  March  and  August  2008  as  well  as  July  and  November  2009.  During 
the  two-year  and  seven-month  period,  the  overall  Excalibur  rating  never  deviated  from 
yellow. 

Selective  Screening  of  Critical  Components 

A  complex  weapons  system  such  as  Excalibur  has  many  moving  parts  and  aspects 
that  can  be  considered  in  a  risk  assessment.  The  Identification  of  Potential  Risk  Areas 
reduced  the  number  of  components  to  consider  to  an  abbreviated  list  that  was  unilat¬ 
erally  system-specific.  To  get  a  better  understanding  of  the  technical  problems  during 
this  period,  a  selective  screening  of  critical  components  was  conducted  to  further  focus 
the  review  on  a  few  of  the  most  important  aspects. 

RAND  developed  the  selective  screening  specifically  to  study  weapons  platforms 
such  as  Excalibur.  The  screening  is  intended  to  limit  the  search  for  challenges,  prob¬ 
lems,  and  failures  to  just  the  most  critical  components.  These  components  are  primary 
aspects  of  the  system  and  are  at  the  core  of  the  programs  function. 

The  selective  screening  of  critical  components  begins  with  two  questions  about 
the  program:  (1)  What  are  the  primary  goals  for  the  platform?  and  (2)  What  technical 
components  will  provide  the  greatest  gains  toward  meeting  the  requirements  for  the 
platform?  Answering  these  questions  for  Excalibur: 

1.  What  are  the  primary  goals  for  the  platform? 

Excalibur  was  designed  to  destroy  key  targets  with  a  minimal  of  rounds,  noncom¬ 
batant  casualties,  and  other  collateral  damage.  Precision  targeting  is  a  primary  goal  for 
the  Excalibur  program. 

2.  What  technical  components  will  provide  the  greatest  gains  toward  meeting  the 
requirements  for  the  platform? 


Figure  5.3 

Data  for  the  Analysis  from  DCMA  Monthly  Reports 
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P.O.  Box  1 1.137, Building 80 1,  M/S  J2 
Tucson,  Arizona  85734-1337 


DCMAM-STRRD 


February  20, 2008 


MEMORANDUM  FOR:  Project  Manager,  Combat  Ammo  Systems,  SFAE-AMO-CAS- 
EX,  LTC  Joe  Minus 

SUBJECT:  Excalibur  Program  Monthly  Report  for  14  January  2008  -  17  February  2008. 

Executive  Summary:  The  overall  DAES  rating  for  this  reporting  period  is  Yellow 
(Potential  or  actual  problems!.  RMS  is  in  the  process  of  delivering  AUR's  for  the  FY07 
contract.  The  program  continues  to  experience  product  risk  issues  with  the  1MU,  GPS  and 
Data  hold  Batteries.  With  the  change  to  the  FY07  contract  and  the  final  delivery  of  the 
FY06  AURs,  the  requirement  for  EVM  reporting  for  the  Excalibur  program  will  be 
complete. 

1.  PROGRAM  DATA 

1.1  Earned  Value:  EV  Program  Analysis  (IEAC)  FY06  ProRram:  An  overall  DAES 
rating  of  Yellow  (Potential  or  actual  problems)  has  been  applied  to  this  section.  There  was 
a  total  of  4  CARs  written  against  Guideline  #6,  #9,  #23  &  #27  as  a  result  of  the  Excalibur 


< 


DCMA  Excalibur  program  monthly 
report  reporting  period  levels  (red, 
yellow,  and  green)  were  coded 
into  a  database  structure. 


Data  were  entered  by  date 


For  the  overall  rating  in  the 
executive  summary 


For  the  program  data  section  rating 


NOTE:  DCMA  color  codes  are  green — no  risk,  yellow — moderate  risk,  and  red — high  risk. 
SOURCE:  DCMA. 
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Figure  5.4 

Excalibur  DAES  Section-Level  Changes  to  the  Risk  Rating 


2008 


NOTE:  CAR  =  corrective  action  request. 
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2009 


2010 
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Excalibur  uses  a  GPS  that  is  aided  by  an  IMU  in  jammed  circumstances.  Using 
triangulated  guidance,  the  canards  glide  or  “fly”  the  projectile  to  the  target.  The  GPS 
and  IMU  enable  Excalibur  to  reach  near-vertical  angle,  achieving  impressive  precision. 

GPS  and  IMU  were  identified  through  the  selective  screening  of  critical  com¬ 
ponents  as  key  aspects  of  the  primary  goals  of  the  platform.  The  following  section 
describes  an  exploration  of  these  components  using  the  DCMA  technical-level  risk 
ratings  data. 

DCMA  Technical-Level  Risk  Ratings 

To  build  a  risk  history  for  the  GPS  and  IMU  systems,  the  DCMA  risk  ratings  and  the 
item  rank  order  values  for  each  report  were  assembled  into  a  research  database.  Figure 
5.5  is  a  snapshot  example  of  the  DCMA  risk  rating  matrix  from  the  monthly  progress 
reports.  The  matrix  contains  the  core  information  necessary  for  the  technical-level  risk 
assessment.  The  DCMA  risk  ratings  in  the  matrix  are  based  on  the  consequence  and 
probability  of  failure  to  meet  the  requirements,  and  the  item  rank  order  indicates  the 
top  failure  concerns — a  ranking  of  #1  is  of  more  concern  than  a  ranking  of  #6.  Taken 
together,  the  ratings  help  to  highlight  the  program  components  of  greatest  concern  on 
a  month-to-month  basis. 

The  DCMA  risk  rating  matrix  data  for  all  of  the  monthly  reports  were  assembled 
into  a  research  database  to  facilitate  identification  of  changes  to  the  valuation  of  spe¬ 
cific  components  so  that  analysts  could  limit  a  detailed  review  of  the  history  and  log 
comments  in  the  monthly  progress  reports  to  specific  components  and  time  periods. 

Because  of  the  limited  amount  of  time  a  decisionmaker  or  reviewing  analyst  may 
have  to  view  and  make  decisions  based  on  all  of  the  available  information,  we  wanted  a 
way  to  consider  the  top-level  flags  alongside  the  corresponding  lower-level  risk  ratings. 
This  was  a  realization  based  on  the  findings  presented  in  Figure  5.4;  the  overall  rating 
never  changed  from  yellow  despite  an  underlying  history  of  lower-level  fluctuations  in 
identified  risk.  Figure  5.6  aligns  the  DAES  (top-level)  and  DCMA  (lower-level)  rating 
scales  along  a  common  time  line  for  the  IMU  to  view  changes  better  and  consider  the 
differences  between  the  ratings  at  specific  time  periods.  The  DAES  ratings  are  in  the 
leftmost  column,  and  the  DCMA  data  appear  in  the  three  right-hand  columns.  The 
IMU  risk  rating  was  raised  to  red  during  the  same  four-month  period,  as  the  section 
rating  deviated  from  the  overall  DCMA  rating  of  yellow  to  red,  depicted  in  Figure  5.6. 
(Note  that  the  oldest  dates  are  at  the  top  and  the  newest  at  the  bottom.)  Although  the 
DCMA  risk  ratings  for  the  IMU  component  are  raised  only  from  moderate  to  high 
from  December  2008  to  March  2009,  a  longer  pattern  of  concern  was  identified  by 
drilling  deeper  into  the  failure  rank  order  of  the  component  (third  column  from  the 
left).  There,  it  becomes  apparent  that  the  IMU  had  experienced  problems  as  early  as 
February  2008. 

The  primary  goal  of  the  work  was  to  explore  why  the  risk  ratings  of  components 
have  changed  over  time.  The  responsive  rating  time  line,  depicted  on  the  far  right  of 


Figure  5.5 

Image  of  the  DCMA  Risk  Ratings  Matrix 


The  program  data 
section  1.4  technical 
performance  was 

coded 


Item  rank  number 


By  DCMA  risk  rating 


1.4.  Technical  Performance  An  overall  DAES  rating  of  YKLLOW  (Potential  or  actual 
problems )  has  been  applied  lo  this  section  because  of  engineering  concerns  as  explained 
below. 


Vn. 

ITEM 

SI  iHIM.il  It 

ST  ATI  K 

DCMA 

Risk  Rating 

Inertial 

Measurement 

UnH 

(IMU) 

Honeywell 

Ongoing  failure  investigations  affecting  the  projectiles  85%  system  reliability 
threshold  (TPM)  Accepting  FY07  IMUs  on  ROWS  that  relax  the  Vibration 
Rectification  Error  (VRE)  gyro  bias  requirements  because  of  failed  QUAL  Tin 
whisker  mitigati jn  plan  established  for  only  half  of  FY07  production  AGO? 
Senes  1  failed  3  out  of  4  rail  gun  tests  white  AG02  Senes  3  failed  1  out  of  4 
rail  gun  tests  AG02  Senes  3  being  utilized  for  FY07  production  while 
engineenng  changes  being  worked  DCMA  predicts  that  overall  system 
reliability  will  be  leopardized  if  future  test  failures  continue  lo  occur 

Moderntc 

#1 

The  test  flight  failures  due  to  loss  of  satellite  acquisition  and  intermittent 
issues  are  affectino  oroiectile  s  85%  system  reliability  threshold  (TPM), 

*2 

GPS  Receiver 

L34EC 

Drivers  mdude  crystal  oscillator  yield  failures  conformal  coat  inconsistencies 
&  test  failures  Failure  Investigations  are  becoming  exhausted  with  no  mot 
cause  after  year  long  expenditure  Failure  team  looking  for  patterns  of  the 
failures  from  multiple  flights  Including  causes  of  frequency  drifts  from  the 
oscillator  DCMA  predicts  that  overall  system  reliability  will  be  jeopardized  if 
future  test  failures  continue  to  occur 

Moderate 

#3 

Dnu*  Hold 
Batteries 

baglc  hdtcr 

Failure  investigation  efforts  conlinue  Program  requested  material  transfers 
from  government  inventory  to  cover  immediate  production  needs  Lot  424  has 

2  cold  nse  time  failures  that  need  waivers  to  allow  shipment  for  lot  #24  and  is 
at  nsk  for  shipment  with  the  program  if  (tvs  lot  is  not  deemed  acceptable  for 
waiver  RMS/EPT  is  currently  testing  different  scenanos  to  resolve  the  lots  in 
question  The  cold  LAT  environment  is  still  an  on  going  risk  DCMA  predicts 
delivery  schedule  slips  from  shortage  of  parts  if  lot  acceptance  tests  (LAT) 
continue  to  fait 

Moderate 

#4 

Propulsion 

Base 

Bolors 

It  is  evident  that  the  failure  mode  is  a  combination  of  events:  unique  test 
conditions,  assembly/rework  process,  and  a  design  margin.  Evidence 
demonstrates  that  there  is  no  credible  nsk  to  the  muzzle  brake  or  the  gun 
crew  per  SSRA  Redesign  efforts  continue  at  Bofors  to  lower  the  probability  of 
future  base  failures  These  efforts  include  increasing  the  strength  of  the 
M40/48  threads,  lower  the  height  of  the  flange  and  increase  the  number  of 
pins  in  the  slipping  driving  band  for  increased  sectional  breakup  within  the 
muzzle  brake.  There  is  a  risk  on  redesign  efforts  meaning  that  fixing  a 
problem  might  create  entirely  new  problems 

Top  Failure  &  Risk  Drivers 
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Figure  5.6 

IMU  DCMA  Technical  Component-Level  Risk  Rating  History 
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IMU  failure  rank 
order  changes 
during  the  cycle 
suggest  that  the 
IMU  challenges 
may  have  started 
earlier 
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associated  with 
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on  the  horizon 
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Figure  5.6,  summarizes  the  major  DAES  and  DCMA  changes  to  the  IMU  risk  ratings. 
On  the  responsive  time  line,  white  reflects  periods  in  which  the  rating  scale  has  not 
changed  and  therefore  nothing  has  occurred  to  improve  or  hinder  the  programs  prog¬ 
ress  toward  meeting  the  requirements.  Periods  colored  red  represent  the  highest  risk  to 
the  program  and  green  periods  the  least. 

By  studying  the  comments  and  other  data  discussed  in  the  monthly  DCMA  pro¬ 
gram  reports,  we  were  able  to  gain  a  better  understanding  of  the  significance  of  the 
risk  rating  change  during  a  specified  time  period.  As  a  result,  the  key  dates  in  the  IMU 
responsive  time  line  indicate  that  problems  with  the  design,  supplier,  and  software  were 
present  well  before  the  DAES  section-level  rating  changed  from  yellow  to  red.  Specifi¬ 
cally,  we  found  failure  investigations  related  to  the  IMU  between  February  and  June 
2008  and  quality/design  issues  with  the  manufacturer  from  the  beginning  of  the  time 
period  in  February  2008  to  February  2009.  These  challenges  represent  serial  patterns 
of  increasing  risk. 

After  reviewing  the  top-  and  lower-level  ratings  as  aligned  to  the  common  calen¬ 
dar  in  Figure  5.6,  the  responsive  rating  time  line  shown  in  Figure  5.7  was  constructed 
so  that  analysts  could  connect  key  dates  to  the  program  office  descriptions  also  found 
in  the  DCMA  monthly  progress  reports.  This  connection  allowed  for  a  more  qualita¬ 
tive  analysis  of  specific  changes  to  the  ratings,  contextualizing  the  root  cause  challenges 
that  persisted  from  one  period  to  the  next.  The  key  dates  used  to  construct  the  respon¬ 
sive  rating  time  line,  shown  in  Figure  5.7,  reveal  that  the  shock  related  to  IMU  risk 
(December  2008  to  March  2009)  occurred  after  a  pattern  of  serial  risk.  Failure  inves¬ 
tigations  of  the  IMU  were  ongoing  from  February  to  June  2008  and  quality  as  well 
as  design  concerns  through  February  2009.  The  shocks  to  the  IMU  progress  are  all 
related  to  failures  or  cracks  in  components  that  occurred  during  testing.  In  Figure  5.6, 
shocks  reflect  increased  risk  levels  during  a  one-  to  two-month  period  (red)  and  serial 
patterns  reflect  risk  with  a  longer  duration  (orange).  Periods  of  interest  are  marked  with 
a  blue  border. 

Through  the  selective  screening  of  critical  components,  the  GPS  was  also  identi¬ 
fied  as  a  system  of  interest.  Similar  to  the  IMU,  the  technical  component— level  DCMA 
risk  rating  matrix  data  for  the  GPS  were  captured  and  analyzed  for  fluctuations  over 
time  using  the  same  methodology  as  for  the  IMU  analysis  (See  Figure  5.8,  which 
depicts  the  GPS  receiver  DAES  and  DCMA  risk  rating  data.) 

Figure  5.8  depicts  an  underlying  pattern  of  DCMA  component  risk  related  to  the 
GPS  receiver.  The  failure  rank  order  data  indicate  that  the  GPS  receiver  was  a  concern 
well  in  advance  of  the  February  shock.  The  DCMA  component-level  and  the  failure 
rank  order  rating  changes  were  used  to  construct  a  responsive  time  line  documenting 
the  history  of  risk  facing  the  component. 

Similar  to  the  IMU,  the  responsive  time  line  for  the  GPS  was  used  to  identify 
time  periods  to  review  more  thoroughly  to  identify  the  cause  of  the  ratings  fluctuation. 


Figure  5.7 

Patterns  of  Risk  Shown  in  the  IMU  Responsive  Time  Line 
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Failure  order  change  due  to  elevation  of  other 
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Figure  5.8 

GPS  Receiver  DCMA  Risk  Ratings  Data 
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Figure  5.9  depicts  the  GPS  responsive  time  line  with  associated  notes  about  perfor¬ 
mance,  failures,  and  technical  challenges. 

In  February  2010,  well  before  the  failed  simulation  and  testing,  the  GPS  experi¬ 
enced  a  series  of  communication  and  software  failures  that  were  present  in  the  data  as 
early  as  February  2008.  Also  shown  on  the  time  line,  problems  with  the  GPS  receiver 
satellite  vehicle  dropout  (a  software-based  problem)  resulted  in  numerous  flight  failures 
in  June  2008.  These  persistent  challenges  resulted  in  lost  time  that  delayed  the  pro¬ 
gram  schedule. 

As  the  risk  ratings  data  are  list-driven,  the  rank  order  of  risks  lead  to  a  change  in 
the  ratings.  As  another  component  rises  or  falls  in  failure  rank  order  importance,  that 
change  affects  the  ratings  of  all  others.  Figure  5.9  captures  an  example  of  this  type  of 
change.  Because  of  an  elevated  risk  of  failure  in  February  2009  that  was  related  to  the 
IMU,  height  of  burst,  and  data  hold  batteries,  the  GPS  rating  changed  from  orange 
to  green  or  from  more  to  less  risk.  This  example  of  change  resulting  from  the  order  of 
other  risks  does  not  dilute  overall  concern  for  the  GPS  receiver. 

Discussion 

Both  the  IMU  and  GPS  receiver  are  technical  components  critical  to  the  overall  func¬ 
tion  of  Excalibur  and  have  been  the  source  of  considerable  risk  over  a  long  period  of  the 
programs  history.  Problems  with  the  IMU  quality,  the  accelerometer,  and  IMU  gyro, 
as  well  as  a  change  in  suppliers,  contributed  to  this  history.  In  the  GPS  receiver  system, 
software  and  integration  challenges  contributed  to  simulation  and  testing  failures. 
Over  the  two-year  and  seven-month  period,  these  persistent  problems  led  to  delays  in 
the  program.  If  the  DAES  section-level  and  DCMA  technical  component— level  risk 
ratings  are  not  followed,  the  persistent  challenges  facing  these  central  components  may 
remain  unnoticed. 

The  experience  in  tracking  the  history  of  technical  problems  faced  by  the  Excali¬ 
bur  program  through  the  DCMA  risk  assessments  informed  the  methodology  depicted 
in  Figure  5.10  to  help  identify  potential  longer-term  risk  to  platforms.  Using  the  appro¬ 
priate  time  series  data  source — in  the  case  of  the  Excalibur  IMU  and  GPS  receiver, 
this  was  the  DCMA  monthly  program  reports — fluctuations  in  the  risk  ratings  are 
captured  on  the  responsive  time  line.  These  points  of  change  from  higher  to  lower  or 
lower  to  higher  risk  are  further  researched  through  the  comments,  figures,  and  other 
data  presented  in  the  monthly  reports.  Conclusions  related  to  each  change  in  the  rat¬ 
ings  are  summarized  along  with  the  responsive  time  line  to  provide  the  analyst  with  a 
sense  of  the  gravity  associated  with  persistent  problems  and  shock  failures. 

The  persistent  problems  should  be  tackled  by  the  analyst  and  program  manag¬ 
ers  first.  These  are  the  components  and  systems  to  inquire  about  regularly  and  to  fully 
understand  before  they  contribute  to  a  shock  failure  event  such  as  the  GPS-related 
simulation  failure  or  failures  with  the  accelerometer  or  IMU  gyro.  The  exploration  of 
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Figure  5.10 

Proposed  Change  Model  Based  on  the  Responsive  Time  Line 
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technical  risks  to  the  Excalibur  program  revealed  a  pattern  of  shocks  and  serial  prob¬ 
lems  that  contributed  to  larger  and  longer-term  complications  for  the  platform. 


Conclusion 

The  RAND-led  root  cause  analysis  of  Excalibur  identified  a  few  specific  points  in  the 
programs  unit  cost  history  that  contributed  to  the  breach.  The  analysis  also  revealed 
several  concerns  related  to  the  potential  technical  risk  in  the  program.  This  chapter  has 
explored  the  technical  risk  further  by  examining  a  series  of  monthly  progress  reports. 
Analysis  of  the  DAES  and  DCMA  risk  ratings  of  technical  components  within  those 
reports  suggests  that  problems  related  to  the  GPS  receiver  and  IMU  were  well  docu¬ 
mented  in  the  more  granular  technical  component-level  risk  data  well  before  red  flags 
were  raised  at  the  overall  technical  level.  Furthermore,  the  investigation  also  revealed 
that  fluctuations  in  the  granular  and  even  the  section-level  data  occurred  without  ever 
raising  red  flags  at  the  overall  Excalibur  program  level.  The  persistence  of  more  granu¬ 
lar  technical  problems  led  to  larger  shock  events,  which  contributed  to  schedule  delays, 
simulation  and  testing  failures,  and  ultimately  to  the  Excalibur  unit  cost  breach. 

As  MDAPs  become  increasingly  complex  and  operationally  more  closely  inter¬ 
related,  new  techniques  must  be  developed  to  enrich  the  decisionmaker’s  and  the 
reviewing  analyst’s  understanding  and  grasp  of  critical  points  of  potential  program 
risk.  This  chapter  introduced  one  approach  to  identify  technical  component-level  risk. 
The  Excalibur  example  described  here  is  not  a  unique  case,  as  the  DAES  and  DCMA 
risk  rating  data  points  studied  were  taken  from  the  reports  generated  regularly  by  the 
defense  acquisition  community.  Other  similar  sources  of  data  covering  a  variety  of 
other  MDAP  performance  measures  exist  within  regularly  generated  defense  acqui¬ 
sition  reports.  The  Nunn-McCurdy  investigations  provide  a  prime  opportunity  for 
these  reports  and  their  rating  scales  to  be  reviewed  and  to  be  better  used.  Reports  that 
are  common  to  the  acquisitions  and  defense  contracting  community  could  be  used  to 
create  new  analytical  methods  for  evaluating  an  MDAP’s  progress  through  all  phases 
of  the  program’s  life  cycle. 


CHAPTER  SIX 


Concluding  Observations 


This  report  has  presented  the  results  of  root  cause  analyses  of  two  programs.  It  has  also 
laid  out  and  illustrated  a  methodology  that  program  managers  could  apply  to  focus 
attention  on  the  elements  of  an  acquisition  program  most  likely  to  lead  to  a  Nunn- 
McCurdy  breach.  To  date,  RAND  has  completed  root  cause  analyses  of  six  programs. 
It  is  important  that  the  perspectives  gathered  through  these  analyses  be  shared  and 
understood  as  we  come  into  the  next  full  season  of  Nunn-McCurdy  reporting.  We 
therefore  offer  our  observations  on  the  conduct  of  RCA. 


Root  Cause  Analyses  of  Nunn-McCurdy  Breaches 

Excalibur 

RAND’s  root  cause  analysis  identified  one  primary  driver  and  four  contributing  factors 
to  Excalibur’s  Nunn-McCurdy  cost  breach.  The  primary  driver  of  cost  increase  was  the 
change  in  procurement  quantities,  specifically,  a  79  percent  reduction  in  Excalibur 
rounds  ordered.  The  root  causes  of  this  quantity  reduction  were  changes  in  require¬ 
ments  combined  with  affordability  considerations.  The  manner  in  which  artillery  was 
being  used  and  the  precision  of  the  Excalibur  round  meant  that  fewer  would  be  needed. 

Four  other  factors  contributed  to  some  program  cost  growth  before  the  decision 
was  made  to  reduce  procurement  quantity: 

•  inaccurate  estimates 

•  concept  and  technological  change 

•  minor  technical  issues 

•  urgent  operational  need. 

Inaccurate  cost  estimates  contributed  to  some  cost  growth.  Both  the  original  May 
1997  cost  estimates  and  the  initial  SAR  estimates  were  too  low  to  reflect  the  techno¬ 
logical  improvement  represented  by  Excalibur,  making  an  eventual  cost  overrun  more 
likely.  Additional  drivers  of  the  cost  growth  before  the  breach  include  a  concept  and 
technological  change  that  occurred  between  the  original  solicitation  and  the  contract 
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award  in  January  1998,  as  well  as  some  minor  technical  issues  between  2002  and  2010. 
Finally,  Excalibur  unit  cost  growth  was  driven  by  the  validated  and  urgent  operational 
need  during  OEF/OIF,  which  caused  production  to  be  accelerated  and  the  production 
of  more  Increment  la  rounds  than  initially  planned. 

Excalibur  was  unaffected  by  other  potential  root  causes.  For  example,  Excalibur 
lived  up  to  its  performance  expectations,  was  not  affected  by  poor  government  or  con¬ 
tractor  performance,  and  had  sufficient  and  fairly  stable  sources  of  funding. 

Navy  Enterprise  Resource  Planning 

Although  the  Navy  ERP  program  technically  breached  the  Nunn-McCurdy  cost 
growth  limits  and  was  implemented  behind  schedule,  the  program  can  still  be  con¬ 
sidered  a  qualified  success.  Most  cost  growth  and  schedule  delays  occurred  in  2004 
and  2005.  Since  the  2006  re-baseline,  costs  have  stabilized  and  production  delays  have 
been  limited. 

The  root  cause  of  the  2004  cost  overrun  stemmed  from  two  sources.  One  was  a 
somewhat  optimistic  baseline  for  cost  and  schedule.  The  second  and  greater  problem 
was  the  unexpected  change  in  business  practices  caused  by  the  Navy’s  decision,  after 
the  BRAC  process,  to  move  maintenance  from  an  intermediate-level  construct  to  a 
regional  one.  The  latter  led  to  the  major  schedule  slippage  in  2005  and  forced  the  ERP 
program  to  jettison  its  extension  to  maintenance  activities. 

The  ERP  program  was  re-baselined  in  2006  at  $400  million  higher.  The  increase 
arose  from  a  redesign  of  the  system,  the  change  in  business  practices,  and  an  improve¬ 
ment  in  estimates.  Since  2006,  ERP  costs  have  stabilized  and  the  program  has  been 
successfully  implemented  at  three  SYSCOMs.  Some  additional  schedule  slippage  has 
occurred  primarily  because  of  timing  issues  rather  than  program  delays  or  failures. 

The  Navy  ERP  can  be  considered  a  qualified  success.  Although  initial  cost  growth 
and  schedule  slippage  were  significant,  they  were  not  extreme,  and  the  ERP  program 
was  never  in  real  danger.  Several  factors  may  have  contributed  to  relative  program  suc¬ 
cess,  including  the  use  of  pilot  projects,  cost-plus  contacting,  the  decision  to  minimize 
the  customization  of  the  SAP  solution,  interactive  governance  and  high  leadership 
interest,  and  a  willingness  to  rely  on  the  managerial  and  technical  expertise  of  civilian 
cadres. 


Program  Complexity  and  Risk 

Our  analysis  of  the  six  programs  that  had  Nunn-McCurdy  breaches  indicates  that  it  is 
possible  to  identify  when  a  program  may  be  at  risk  of  a  breach  far  sooner  than  has  been 
the  case  in  the  past.  Not  all  aspects  of  a  program  have  the  same  potential  to  lead  to  a 
breach.  The  challenge  for  program  managers  is  to  focus  on  the  elements  of  the  program 
most  likely  to  lead  to  a  breach.  Two  attributes — level  of  complexity  and  detail — can 
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help  identify  those  elements.  The  components  of  a  program  that  are  especially  complex 
and  require  a  high  level  of  detail  are  the  ones  that  require  the  most  attention. 

By  identifying  those  elements  early  and  focusing  on  them,  a  program  manager 
can  uncover  early  indications  of  when  these  elements  may  be  leading  to  higher  pro¬ 
gram  risk.  MDAPs  have  voluminous  documentation.  Using  complexity  and  level  of 
detail  to  identify  program  elements  that  lie  along  the  critical  path  to  success,  program 
managers  need  to  delve  into  the  documentation  of  elements  for  signs  of  risk.  This 
means  going  well  below  the  summary-level  documentation,  because  in  the  aggregation 
of  ratings  to  the  system  level,  important  details  can  be  missed. 


Observations  on  the  Conduct  of  an  RCA 

Many  sources  of  data  must  be  consulted  to  allow  the  RCA  team  to  compile,  under¬ 
stand,  and  interpret  key  events  in  a  program’s  history,  from  military  requirements  and 
financial,  technical,  contractual,  schedule,  and  acquisition  environment  perspectives. 
Key  sources  of  data  to  attain  this  knowledge  include  ADMs,  acquisition  strategies, 
Cost  Analyses  Requirements  Description,  DAES,  DCMA  reports,  information  gained 
in  interviews,  formal  letters  of  notification  to  Congress  of  Nunn-McCurdy  breaches, 
Nunn-McCurdy  Overarching  Integrated  Process  Team  cost  and  management  brief¬ 
ings,  other  official  program  briefings,  PDRs,  SARs,  and  OUSD  (AT&L)  program 
memoranda.  Numerous  secondary  sources  of  data  also  must  be  consulted  to  verify 
data  and  solidify  insights  offered  by  primary  data  sources.  Interviews  with  program 
and  industry  officials  are  also  valuable  sources  of  information  and  provide  insights  that 
are  difficult  to  gain  from  only  documentation.  The  precise  combination  of  documents 
and  interviews  appropriate  for  each  RCA  evolves  during  the  course  of  conducting  the 
analysis.  Since  repeat  consultations  with  various  sources  are  part  of  the  RCA  process, 
the  data  sources  should  be  recorded  in  bibliographical  form  for  inclusion  in  the  formal 
report  and  in  searchable  form  for  future  consultation. 

Although  each  program  is  unique  and  thus  each  RCA  is  unique,  a  set  of  core 
activities  is  instrumental  to  a  successful  root  cause  analysis.  These  activities  define  a 
generic  root  cause  methodology  whose  key  components  include  the  following: 

•  Develop  a  hypothesis. 

•  Set  up  long-lead-time  activities. 

•  Document  the  unit  cost  threshold  breach. 

•  Construct  a  time  line  of  cost  growth  relevant  events  in  the  program  history. 

•  Verify  the  cost  data  and  quantify  cost  growth. 

•  Create  and  analyze  the  program  cost  profiles  pinpointing  occurrences  cost  growth. 

•  Match  the  time  line  events  with  the  changes  in  the  cost  profiles  and  derive  root 
causes  of  cost  growth. 
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•  Reconcile  remaining  issues. 

•  Attribute  unit  cost  growth  to  root  causes. 

Execution  of  the  above  activities  should  result  in  enough  useful  material  for  an 
RCA  team  to  create  the  primary  outputs  supporting  a  root  cause  analysis:  a  summary 
narrative  that  includes  clearly  stated  root  causes  of  cost  growth  supported  by  a  formal 
documentation  of  the  cost  threshold  breach,  a  summary  time  line  of  program  events 
leading  to  the  Nunn-McCurdy  breach,  funding  profiles,  a  completed  PARCA  office¬ 
generated  root  cause  matrix,  and  a  breakdown  of  the  amount  of  cost  growth  attribut¬ 
able  to  each  root  cause;  an  informal  briefing  incorporated  into  the  final  report;  and  a 
full  root  cause  final  report. 

RAND  has  conducted  six  RCAs  for  the  PARCA  office.  These  efforts  have  gen¬ 
erally  involved  three  to  four  personnel  working  rapidly  for  an  average  of  35  days.  As 
described  above,  vital  to  the  success  of  these  efforts  was  timely  access  to  a  wide  variety 
of  program  documentation  and  people  involved  in  the  program.  A  companion  pub¬ 
lication  associated  with  this  project  arrays  the  sources  in  a  different  manner  to  fully 
explore  both  the  nature  of  the  data  available  for  use  and  the  source  of  various  data  in 
addition  to  the  data  found  from  more  customary  approaches.1  This  list  of  sources  is  an 
important  one  to  appreciate  fully  because,  although  some  data  can  be  gathered  from 
certain  sources  in  anticipation  of  a  breach  occurring,  some  significant  quantity  of  data 
can  be  gathered  only  after  the  secretary  of  the  cognizant  military  department  notifies 
Congress  of  the  Nunn-McCurdy  breach.  The  discussion  and  graphs  contained  in  the 
companion  report  fully  support  the  ability  to  make  distinctions  as  to  the  categories  and 
sources  of  data  as  they  become  available  for  users.  Analysts  preparing  to  examine  these 
potential  and  notified  breaches  need  to  be  aware  of  the  distinctions  to  be  able  to  set  up 
the  long-lead-time  activities  addressed  earlier  in  our  generic  methodology. 

It  is  important  to  remember  that  each  program  is  unique  and  the  sequence  of 
events  that  lead  to  a  program’s  unit  cost  threshold  breach  will  be  unique  as  well. 
Although  this  report  describes  the  basic  data  sources  that  should  be  consulted  and  a 
generic  methodology  for  successfully  conducting  a  root  cause  analysis,  further  RCAs 
may  require  only  part  of  the  methodology  and  only  some  data  sources,  whereas  others 
may  require  all  that  is  described  plus  much  more.  Hence,  to  conduct  an  RCA  within 
the  legally  allotted  time  frame,  analysts  will  need  to  be  diligent  in  pursuing  sources  of 
information  that  can  lend  insight  to  the  analysis. 

The  last  two  analyses  conducted  by  RAND  exemplify  this  need  for  resourceful¬ 
ness  and  adaptation.  For  Excalibur,  the  data  available  and  provided  initially  were  not 
useful  in  answering  certain  questions  dealing  with  program  status.  It  was  only  after  the 
RCA  was  largely  complete  that  certain  data  were  captured  and  examined  more  fully. 
These  data,  developed  by  DoD  on  an  ongoing  basis,  if  viewed  correctly,  would  enable  a 
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See  Blickstein  et  al.,  forthcoming. 
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continuing  risk  analysis  of  programs  by  acquisition  leaders.  The  data  supporting  a  risk 
analysis  were  not  gathered  easily  or  even  for  all  programs  examined  by  RAND,  and 
this  continues  the  experience  addressed  in  Blickstein  et  ah,  2011. 

Similar  to  the  cost-based  approach  to  the  Excalibur  RCA,  the  risk  analysis  of 
technical  components  revealed  that  challenges  early  in  the  program’s  history  increased 
the  risk  of  not  meeting  requirements.  The  approach  described  in  Chapter  Five  used 
a  responsive  rating  time  line  to  track  the  performance  of  several  components  critical 
to  Excalibur  and  found  that  as  early  as  February  2008,  the  GPS  receiver  and  IMU 
were  emerging  concerns.  Although  problems  with  the  IMU  accelerometer  triggered 
a  higher-level  risk  rating  at  the  end  of  2008,  problems  related  to  the  GPS  remained 
obscured  in  component  details  until  simulation  and  testing  failures  during  February 
2010.  The  approach  outlined  in  this  chapter  is  provided  to  help  identify  a  methodol¬ 
ogy  to  highlight  questionable  performance  before  a  failure.  Although  there  needs  to  be 
a  standard  for  data  supporting  MDAP  oversight,  it  may  well  be  that,  given  the  nature 
or  source  of  the  programs  considered,  some  special  approaches  such  as  that  presented 
here  may  be  useful. 

Critical  to  assembling  the  responsive  rating  time  line  used  in  Chapter  Five  is  the 
selective  screening  of  critical  components  exercise  described  in  Chapter  Four.  Comple¬ 
tion  of  the  RCAs  reported  in  this  document  led  to  a  clearer  understanding  of  the 
need  to  focus  the  review  of  MDAP  material  early  in  any  analysis.  This  realization  has 
become  a  cornerstone  to  the  methodology  presented  in  Blickstein  et  al.,  forthcom¬ 
ing,  and  Chapter  Four  proposes  that  this  type  of  exercise  might  also  be  useful  for 
decisionmakers  before  a  program  triggers  a  Nunn-McCurdy  breach.  The  authors  felt 
that  emphasis  on  the  complex  and  least  visible  components  of  a  program  would  be 
most  useful,  though  clearly  other  frameworks  can  be  devised.  The  exercise  described 
in  Chapter  Four  identifies  the  components  of  most  interest  to  the  decisionmaker,  and 
the  methodology  outlined  in  Chapter  Five  describes  the  depth  of  analysis  necessary  to 
reveal  underlying  risks.  The  strategies  discussed  will  help  a  decisionmaker  design  an 
inquiry  strategy  for  uprooting  risk  from  complex  programs. 

Our  examination  of  our  last  two  programs  also  provided  insight  that  led  to  the 
designation  of  another  approach  in  examining  program  execution  for  consideration. 
First  with  Excalibur  and  then  again  with  Navy  ERP,  understanding  the  fundamental 
characteristics  of  the  requirements  relationships  of  programs  entering  into  an  MDAP 
or  other  senior  review  and  oversight  process  appears  to  be  essential  to  both  fuller  over¬ 
sight  as  well  as  more  complete  analysis.  With  Excalibur,  one  observation  is  that  one 
size  does  not  fit  all.  Methodologies  that  look  at  all  programs  the  same  way  fail  to  look 
at  programs  in  necessary  ways.  In  this  report,  Excalibur  is  explored  at  the  macro  level 
through  the  root  cause  analysis  presented  in  Chapter  Two  and  at  the  granular  level  in 
Chapter  Five.  This  was  necessary,  partly  because  of  Excalibur’s  development  history  in 
relation  to  other  systems  and  partly  as  a  defense  system  unlike  other  programs  assigned 
to  RAND  to  date  for  review. 
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The  programmatic  relationship  between  the  Army’s  sweeping  modernization  pro¬ 
gram  Future  Combat  System  and  Excalibur  was  known.  That  relationship  had  embed¬ 
ded  into  it  certain  requirements  and  technical  interrelationships  that  may  well  have 
been  central  to  the  ultimate  success  of  Excalibur.  With  the  demise  of  Future  Combat 
System,  a  fuller  examination  and  understanding  of  the  implication  for  Excalibur  might 
have  led  to  a  better  understanding  of  quantity  issues.  This  greater  understanding  may 
have  led  to  earlier  decisions  and  a  re-base  to  the  assumptions  tied  to  the  moderniza¬ 
tion  architecture  in  addition  to  the  individual  programs  realities.  To  the  extent  that 
DoD  has  similar  situations  with  various  overarching  modernization  architectures,  it 
may  well  serve  a  useful  purpose  to  engage  in  architecture-oriented  reviews.  DoD  is 
currently  suggesting  that  quantity-related  breaches  be  waived  from  Nunn-McCurdy 
provisions.  Care  should  be  taken  in  pursuing  this  initiative  so  that  more  central  issues 
than  mere  quantity  reduction  are  not  masked. 

In  the  case  of  Navy  ERP,  we  were  able  to  see  in  the  process  of  the  RAND  review 
that  the  central  characteristic  of  the  program  was  the  re-engineering  of  business  pro¬ 
cesses,  many  of  them  dating  to  World  War  II.  The  recognition  of  this  characteristic  is 
central  to  oversight  and  analysis  of  such  a  program.  Its  implications  appear  to  be  insuf¬ 
ficiently  (if  at  all)  understood  by  higher  authority.  The  lack  of  understanding  reinforces 
our  view  of  the  need  for  clarity  in  understanding  the  often  complex  relationships  and 
characteristics  that  modern  programs  entail.  Understanding  the  relationship  of  pro¬ 
cesses  being  re-engineered,  by  both  in-house  and  government  and  contractor  personnel 
in  supporting  activities,  to  the  software  being  procured  from  outside  vendors  is  critical 
to  understanding  the  nature  of  cost  estimates  and  program  performance.  As  a  result 
of  the  understanding  gained  across  all  six  of  the  programs  reviewed  and  as  particularly 
illuminated  by  the  difficulty  in  analyzing  Excalibur  and  Navy  ERP  using  traditional 
program  review  approaches,  RAND  developed  an  in-house  template  for  understand¬ 
ing  the  requirements  relations  preposition.  That  template  can  be  found  in  Chapter 
Five.  It  is  provided  to  help  the  PARCA  office  explore  new  and  possibly  more  useful 
approaches  going  forward. 
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